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Key points 


Australia's position 

Australia's universities are doing reasonably well on the published world rankings. That is a cause 
to celebrate because it signals to the world that we have several high guality universities whose 
presence underpins a national system of generally good guality higher education. Being placed 
among the top universities in the world on a range of rankings has reputational and strategic 
positioning advantages for individual universities and their home countries, and status benefits 
for the students and alumni of institutions. 

While the specific positions of universities on the rankings may vary, the clustering of the 
same institutions within the top band on multiple measures cross-validates a leading group of 
Australian universities. From these various sources it is evident that while there are changes in 
the position of some universities on different measures there is a common group of around 20 
Australian universities positioned in the top 1,000 on several of the measures. All of the Australian 
universities in the top 200 on any measure are Go8 universities. Field-specific rankings reinforce 
the impressions of the major institutional rankings but also reveal greater degrees of variation 
among institutions. 

Australia's leading universities position reasonably well against the US land-grant public four-year 
colleges, the UK's Russell Group, Ganada's U1 5, and the top universities of Europe and Asia. 

The danger of complacency 

The indicators used in all of the major rankings are necessarily lagged. Many Asian institutions are 
recipients of substantial increases in government investment in higher education and university 
research. The rate of growth in academic publications output from Asia is far outstripping that of 
Australia and the guality of Asia's research outputs is rapidly improving. 

In contrast, several former leading universities in the US and the UK are experiencing the effects of 
disinvestment as government funding falls and annual returns from university funds under investment 
deteriorate. The shift of relative input strength in favour of Asian universities will show through first in 
the rankings that make use of input measures and later in those that feature output measures. 

Efforts by the university rankers to improve their methods, including efforts to reduce country 
bias in reputational surveys, and to expand the country markets for their rankings, suggest that 
the earlier dominance of universities of the developed economies of North America, Britain and 
Europe will decline over time. Australian universities may appear to be improving against this 
backdrop of decline, whereas they may well be slipping behind the emergent Asian leaders. 


The underlying concept of rankings 

University rankings fill a void. Simple guides are wanted by people having to make choices faced 
with an ever increasing complexity of university functions, organisational arrangements and a 
proliferation of data including marketing hype. The rankings are a means to serve commercial ends 
- both for the rankers and the ranked, but with the latter captured by the imperative of upholding 
and enhancing reputation, the currency of academic value. Rankings appear to be based on 
'objective' assessments which are independent from the universities being ranked. They resonate 
in an academic culture of fierce individual competition and status seeking. 
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World university rankings in particular are selective and norm-referenced within a zero-sum game. 
Rankings are also reductionist and disproportionately value what is comparably measureable 
on the basis of available data. They have not only a descriptive or analytical role but a normative 
function in assigning universities in vertical relation to one another without accommodating 
horizontal diversity. 

Even if the rankings suggest something about elite institutions in the top 3% of the world's 1 7,000 
higher education institutions, they suggest nothing about the relative claims of the other 97% 
and the criteria by which they ought to be judged. Indeed, the 'exclusion' of the majority, or their 
implied relegation, generates inappropriate pressures for them to emulate those that are ranked 
and to distort or compromise their own missions in so doing. 


Methodological deficiencies of the rankings 

The rankings are largely shaped by the limited availability of comparable data. The rankings have 
a validity problem - the lack of alignment between the evaluative criteria and the measures used 
- which is most troublesome for comparisons relating to teaching and learning. The data sources 
for compiling some of the indicators used to derive world university rankings do not capture 
important outputs of different universities in different fields. 

The weights assigned to different indicators within the rankings are arbitrary. There is also no 
correction for redundant measures - the interactions among the variables. There is no standardisation 
of the distribution of scores. Several of the indicators in some of the rankings are prone to subjectivity 
and manipulation. The processes by which the survey data are collected and reported are 
opague.The volatility of the rankings reflects their lack of statistical reliability, arising from poor 
aggregation, without prior standardisation, of different performance indicators for the rankings. 

An important implication of these methodological concerns is that shifts up or down in 
institutional positions do not necessarily reveal substantive changes in institutional performance. 

Institutional responses to rankings 

The major uses of rankings are: 

i. influencing consumer choice, including information and guidance for prospective students, 
and institutional marketing 

ii. benchmarking for improvement, including setting targets and introducing process changes 
based on practices of highly-ranked institutions 

iii. partner selection, including recognition of gualifications or prior study for credit, and choice 
of institutions for collaboration in research and research training 

iv. informing investment decisions, including institutional actions to attract industry and 
benefactor funding, and government decisions to allocate funds for special programs and 
centres or to skew allocations through performance incentives linked to rankings. 

Some universities have dedicated staff for managing their rankings but one university's position 
on the rankings cannot be managed simply by its own efforts. This is because of constant change 
in the operating environment, the efforts of others to improve their positions, and aspects of the 
rankings tools themselves - both changes in methods of counting and inherent methodological 
deficiencies - which render year-on-year results volatile and unpredictable. 
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Public policy implications of rankings 

World university rankings have given rise to three relatively new phenomena in public policy: 
governments targeting the availability of scholarships for their nationals to highly ranked 
universities internationally; governments limiting the partnering of their home universities to only 
those foreign universities that are highly ranked; and governments aspiring to have a number 
of their universities highly ranked or otherwise designated 'world class'institutions through 
concentrated investments or structured collaborative arrangements, such as clusters and mergers. 

As they are currently structured, world university rankings do not relate well to the missions of 
universities whose principal mission is not research, or at least not internationally-referenced basic 
research. There are important roles for universities that focus on producing guality graduates for 
Australian labour markets and providing services, including research translation, testing, survey 
and other services for businesses and community groups. In this regard, institutional positioning 
strategies pulled by the current suite of world university rankings which privilege academic 
research can limit institutional vision and distort mission, and reinforce sameness in aspiration and 
benchmarks of success. 

An important challenge is to develop diverse reference points. For that to happen, three developments 
will be necessary. First, there will need to be sharper clarity of mission orientation on the part of 
different universities and other higher education institutions. Second, work will need to be done 
on developing a richer array of education and service oriented indicators of university capacity and 
performance. Third, attention will need to be given to describing the array of research orientations. 

Australia's relative improvement in several of the world university rankings reflects the plateauing 
of inputs to US and UK universities, the lagged accounting for the emergent Asian universities, 
alongside some recent absolute lifts in funding inputs for Australian universities. 

The apparent rise of some Australian university positions on the rankings cannot be attributed 
simply to increased government funding in recent years. A major capital injection was made 
into research infrastructure in 2009-1 0, the indexation of Commonwealth payments has been 
improving since 2009 and there had been an important increase in funding the indirect costs of 
research, but that has now stalled. Given the lagged nature of rankings indicators, current positions 
reflect the previous Government's doubling of funding for competitive grants administered by the 
Australian Research Gouncil and the National Flealth and Medical Research Gouncil, and growth 
in revenue from international fee-paying students, alumni and industry. Funding for the Research 
Training Scheme (RTS) funding per Domestic FIDR EFTSU has fallen by over 23% between 2001 and 
201 2. Block funding through the Institutional Grants Scheme/Joint Research Engagement scheme 
(IGS/JRE) has declined in real terms over the same period. 

The ERA evaluative process itself from 2009 and subseguently has provided incentives, 
reputational as well as financial, for universities to address areas of weak research performance 
and build on their research strengths. This sharper focus on research performance may well have 
influenced factors taken into account by the world university rankings. 

Eor the global reputation of Australian universities to be sustained, within a more diversified higher 
education system responsive to varying domestic student needs, there will need to be a sharper 
focus on building on our demonstrable strengths. Diluting the national investment in research will 
inevitably see Australia falling off the international league ladder. 

Australian universities can hold or improve their ranked positions chiefly by strengthening their 
capacity, performance and visibility in long-term basic and applied research, especially by raising the 
volume of high guality, highly cited research outputs, including through international co-authoring. 
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The world university rankings use research metrics linked to the outputs of basic research. On 
the one hand, concerns arise that the pressure to align institutional behaviours with the rankings 
measures will lead to a reduction of more socially relevant and commercially useful research. Such 
concerns imply a need to develop indicators to capture the contributions to innovation through 
human capital development and knowledge generation and application that are not captured by 
the metrics used in the current set of world university rankings. On the other hand, we are seeing a 
convergence internationally in a public policy emphasis on the utility of publicly-funded research, 
its demonstrable 'impact'and commercialisability, and some spurning of basic research. 

Whereas in 1 969 basic research represented 76.7% of university research in Australia by 2010 it 
had fallen to 45.2%. Over the same period applied research increased its share from 1 9.7% to 
46.7%, making it the dominant type of activity. In contrast, the basic research share of all research 
conducted in US academic institutions has been reasonably stable at around three guarters over 
the same period (76.7% in 1970 and 74.6% in 2009). 

The combination of inadeguate investment, social levelling and narrow instrumentalism 
in policy for higher education and university research is inimical to sustaining internationally- 
benchmarked excellence. 


The rankings phenomenon 

Arguably it all started in 1 925 when Professor Raymond Hughes, the then President of Miami 
University in Ohio, initiated a ranking of US graduates schools on the basis of the reputational 
judgements of a selected group of academic peers across 20 disciplines in 36 institutions. 

However, it was not until 1 983 that the first ranking of national institutions was undertaken by 
the US News and World Report, making inter-institutional comparisons based on metrics derived 
largely from administrative data reported by institutions for public accountability purposes. 
Macleans Guide to Canadian Universities rankings began in 1991, and the annual issue of that 
publication which includes the rankings now makes sufficient profit to sustain the entire journal. 

In Australia, The Good Universities Guide, currently published by Hobsons, was initiated in 1 991 by 
Dean Ashenden and Sandra Milligan but it chose to rate rather than rank national institutions and 
provided indicators by field of education as well as at the whole of institution level, drawn largely 
from the accountability-based administrative data sets of the central government and with a 
focus on the teaching function of universities, including student access and eguity, retention and 
completion, student satisfaction and graduate destinations and starting salaries. 

It is the rise of global or'world rankings' that has gained most attention since the appearance of 
the Asiaweek magazine's rankings in 2000, the Swiss Centre for Science and Technology's 2002 
'champions league' of research universities, and the Shanghai JiaoTong rankings in 2003 (later to 
be called Academic Ranking ofWorld Universities - ARWU). The latter captured a surprising level of 
attention, not least because its methodology was based on objective research-related indicators 
and the ranking of the top 40 or so institutions had reasonable face validity, even if skewed in 
favour of US institutions by virtue of their scale of post-World War II Nobel Laureates and highly 
cited publications (see Table 1). 

In 2004, an alternative set of rankings was published by the London Times Higher Education 
Supplement with a view to providing a European (or at least a British) counterbalance to ARWU by 
including non-research indicators such as in-bound international students and reputational survey 
data. These rankings were published until 2009. In 201 0 the Times Higher broke off its relations 
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with Quacarelli Symonds over concerns about bias in its methodology (e.g. low response rates 
from its surveys, and under-reporting of Humanities and Social Sciences disciplines). QS continued 
its rankings as a separate exercise but adopted Elsevier's Scopus data base. The Times Higher 
created a new method with Thomson Reuters, using the Thomson ISI data base alongside a wider 
range of reputational and other indicators. In 2008 another set of research-related indicators for a 
range of disciplines had been initiated by Leiden University. As shown in Table 1 , there Is a general 
convergence of the top ranked group of universities, alongside some curious deviations even 
within that group, e.g. Cambridge ranks vary from 2 to 31, and Rice from 4 to 139. 


Table 1. Non-Australian universities on the five main world university rankings, 2012 


University 

Academic 
Ranking of World 
Universities 
Shanghai JT 2012 

World University 
Rankings 
THE 2012-13 

QS World 
University 
Rankings 
2012 

Leiden Ranking 
2011/12 

Webometrics 
July 2012 

Harvard 

1 

4 

3 

3 

1 

U Cal Berkeley 

4 

9 

22 

8 

4 

Stanford 

2 

2 

15 

5 

3 

MIT 

3 

5 

1 

1 

2 

Cambridge 

5 

7 

2 

31 

20 

CalTech 

6 

1 

10 

6 

24 

Princeton 

7 

6 

9 

2 

35 

Columbia 

8 

14 

1 1 

17 

12 

U Chicago 

9 

10 

8 

14 

18 

Oxford 

10 

2 

5 

36 

25 

Imperial 

24 

8 

6 

54 


Yale 

1 1 

1 1 

7 

1 1 

16 

UCL 

21 

17 

4 

59 

43 

U Pennsylvania 

14 

15 

12 

22 

7 

Rice U 

91 

79 

120 

4 

139 

U Cal Santa Barbara 

34 

35 

118 

7 

49 

Carnegie Mellon 

51 

22 

49 

9 

19 

Cornell 

13 

18 

14 

40 

5 

Johns Hopkins 

17 

16 

16 

38 

53 

ETH Zurich 

59 

12 

13 

18 

29 

U Michigan 

22 

20 

17 

30 

13 

Duke 

36 

23 

20 

28 

28 

Edinburgh 

51 

32 

21 

66 

64 

U Toronto 

27 

21 

19 

87 

32 

McGill 

63 

34 

18 

140 

74 

U British Columbia 

39 

30 

51 

103 

35 

University of Hong Kong 

151-200 

35 

23 

182 

108 

National University of Singapore 

101-150 

29 

25 

83 

75 

U Auckland 

151-200 

161 

82 

299 

115 


There have been some interesting shifts also in the ranked position of various institutions over 
time. According to the QS metrics, with only a few exceptions US institutions have fallen at the 
expense of British, European, Aslan and Australian institutions over time. UC Berkeley fell from 2 
In 2004 to 22 in 201 2 and Stanford from 7 to 1 5. Berkeley's fall is attributed to input factors: its 
relatively unfavourable faculty/student ratio and relatively low proportion of international students 
and faculty (Ince, 201 2). The former reflect fiscal constraints in the State of California and the latter 
is a conseguence of the rapid internationalisation of higher education in many countries, except 
for the US until very recently. 
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Importantly, albeit regrettably, when institutions have looked good on the rankings they have 
made unashamed marketing use of them, even where they know at least some of them to 
be guestionable.That usage, whether on billboards on the approaches to major airports or 
in university media statements, brochures and advertisements, has given even the dubious 
rankings a veneer of respectability. More than that, such usage has generated pressure for others 
not so highly ranked to emulate those at the top. While there seems no limit to the creativity 
of universities to position themselves at or near the top of some set of rankings, the sudden 
dominance of a few global rankings based on a narrow range of metrics has forced many 
universities to assess themselves against those metrics and set targets for raising their scores 
even where it is inappropriate and unreasonable for them to do so. The problem is, however, that 
there are few available metrics, beyond a basket of research indicators, which can be used for the 
purposes of international comparisons of universities. This fundamental deficiency in the context 
of increasing reference to the published rankings puts undue pressure on universities whose 
missions are less research oriented and makes more urgent the need to develop a wider and more 
balanced set of indicators. 

Interestingly, the ARWU rankings developed by the School of Education at the Shanghai Jiao 
Tong University (one of the universities given special funding by the Government of China under 
its 985 and 21 1 programs in an effort to raise the standing of China's leading universities to 
'world-class' status) were initially conceived as a means of exposing how large a gap China had 
to close to achieve the Government's goals of having world-class universities (Liu, 2009). That is, 
those rankings were intended as a body of evidence to caution against unrealistic expectations. 
Ironically, they have become a vehicle for universities and governments around the world to raise 
their aspirations, in many cases, unrealistically. 

Today there is a proliferation of mostly commercial guides, including national and international 
rankings, many of which report on whole institutions and some on parts of institutions or 
disciplines (see Table 2 for an indicative but by no means exhaustive list). As Simon Marginson has 
noted, the number of university rankings is expected to continue increasing but (hopefully) they 
will become more specialised (Marginson, 201 1). 


Table 2. National and international rankings of universities, disciplines and departments 


Institutional 

International 


Leiden Ranking - Centre for Science and 
Technology Studies (CWTS) (Netherlands) 
Performance Ranking of Scientific Papers for 
Research Universities [HEEACT] (Taiwan) 
Professional Ranking of World Universities 
(France) 

SCImago Institutional Rankings 
Academic Ranking of World Universities 
[ARWU] (China) 

QSTop University (UK) 

THEThomson Reuters World University 
Rankings (UK) 

U-Multirank (European Commission) 
Webometrics (Spain) 


Discipline/Sub-Categories Specialist 


• Business Week MBA 


• Economist Intelligence 
World MBA Rankings 

• Financial Times MBA 

• Wall Street Journal MBA 


University Systems 
Ranking. Citizens and 
Society in the Age of 
Knowledge (Lisbon 
Council) 

National System 
Strength Rankings (QS) 

Green Metric World 
University Ranking 
(Universitas Indonesia) 
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Institutional 

Discipline/Sub-Categories 

Specialist 

National 

• Chamber of Commerce and Industry 

• Asiaweek MBA School 

• Saviors of Our Cities (US) 

(Sweden) 

Rankings (2000) 

• Washington Monthly 

• CHE-HochschulRanking (Germany) 

• Brian Leiter's Law School 

College Guide (US) 

• Forbes College Rankings (US) 

• Good University Guide (Australia) 

• Google College Rankings (Various) 

• Guangdong Institute of Management 
Science (China) 

Rankings (US) 

• Dataquest (India) 

• India Today (India) 

• Le NouvelObservateur 
(France) 

• Washington Monthly 
Ranking of Community 
Colleges (US) 

• Which? magazine (UK) 

• Guardian University Guide (UK) 

• Mint (India) 


• La Republica (Italy) 

• Outlook (India) 


• Macleans On Campus (Canada) 

• Sharif Magazine (Iran) 


• National Accreditation Centre Rankings 

• Toplawschools.com (US) 


(Kazakhstan) 

• Undergraduate 


• Netbig (China) 

• OHEC (Thailand) 

• Perspektywy (Poland) 

American universities 
rankings for 
international students 
(US) 


• Petersons College Rankings (US & Canada) 

• USNWR Top Med 


• Princeton Review (US) 

Schools (US) 


• Sunday Times (Ireland) 

• Times Higher Education University Guide 
(UK) 

• University Rankings (Ukraine) 

• U-rank (Sweden) 

• USNWR College Rankings (US) 

• Washington Monthly (US) 

• WPROST MBA (Poland) 


• Wuhan University Research Centre for 
Science Evaluation (China) 



Regional 

• AsiaWeek - Asia's Best Universities 
(Hong Kong) 

• CHE Excellence Ranking Graduate 
Programmes 

• Ranking Iberoamericano (Pan Hispanic) 


Source: Adapted from Hazelkorn, 2011. 


The focus of this paper is on the five major global university rankings: 

• Academic Ranking of World Universities (ARWU) - of 500 universities - produced and published 
by a group of education researchers at Shanghai JiaoTong University 

• T/-/f Thomson Reufers 1/l/orW Un/Vers/ty /?on/c/ngs - of 400 universities - published by the London 
Times Higher Education Supplement 

• QS World University Rankings - of 700 universities - produced by Quacguarelli Symonds and 
published by US News & World Report and Chosun llbo 

• Leiden Ranking - of the scientific performance of 500 universities - produced and published by 
the Centre for Science and Technology Studies at Leiden University, the Netherlands 

• Webometrics - of all higher education and research institutions worldwide - produced twice a 
year by the Cybermetrics Laboratory of the Spanish National Research Council. 

The methodological approaches of each of these rankings are outlined in Appendix 1. 
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The position of Australian 
universities in world rankings 

On the five main world rankings of universities Australia has variously 5, 6, 7, 0 and 2 in the top 
1 00. On those same measures Australia has 1 9, 1 8, 25, 1 4 and 1 6 in the top 500. In the top 1 00, 
Australia's performance ranges from 0% to 7%. In the top 500, Australia's performance ranges from 
2.8% to 5%. Except for the Leiden ranking, Australia does relatively better at the top end of the 
international distribution on the measures reflected in the rankings. 

All of the Australian universities in the top 200 on any measure are Go8 universities. While the 
specific positions of universities on the rankings may vary, the clustering of the same institutions 
within the top band on multiple measures cross-validates a leading group of Australian 
universities. Table 3 shows the position of ranked Australian universities on the most recent 
editions of the main world rankings. 


Table 3. Australian universities on the five main world university rankings, 2012 


University 

Academic 
Ranking 
of World 
Universities 
(Shanghai Jiao 
Tong) 201 2 

World 

University 

Rankings 

THE-Thomson 

Reuters 

2012-13 

QS World 
University 
Rankings 
2012-13 

Leiden Ranking 
2011/12 

Webometrics 
July 2012 

ANU 

64 

37 

24 

1 14 

84 

U Melbourne 

57 

28 

36 

163 

101 

U Sydney 

93 

62 

39 

290 

117 

U Queensland 

90 

65 

46 

170 

97 

UWA 

96 

190 

79 

306 

324 

Monash 

101-150 

99 

61 

261 

109 

UNSW 

101-150 

85 

52 

250 

116 

U Adelaide 

201-300 

176 

102 

314 

255 

Macquarie 

201-300 

251-275 

233 

202 

337 

Flinders 

301-400 

351-400 

342 

NR 

666 

James Cook 

301-400 

NR 

362 

NR 

681 

Newcastle 

301-400 

276-300 

268 

382 

461 

Wollongong 

301-400 

301-350 

264 

233 

597 

Curtin 

401-500 

NR 

258 

NR 

609 

Latrobe 

401-500 

NR 

375 

NR 

639 

Swinburne U T 

301-400 

NR 

496 

NR 

1044 

U Tasmania 

301-400 

351-400 

357 

378 

470 

QUT 

NR 

251-275 

281 

367 

398 

Charles Darwin U 

NR 

301-350 

NR 

NR 

1474 

Deakin 

NR 

351-400 

417 

NR 

445 

Griffith U 

301-400 

NR 

368 

412 

619 

UniSA 

NR 

NR 

293 

NR 

459 

RMITU 

NR 

NR 

246 

NR 

387 

UT Sydney 

401-500 

351-400 

284 

NR 

470 

Bond 

NR 

NR 

380 

NR 

1688 

Murdoch 

NR 

301-350 

448 

NR 

793 

U Western Sydney 

NR 

NR 

618 

NR 

640 


NR = Not Ranked 
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Australian universities occupy varying positions on other rankings too. Table 4 shows Australian 
universities ranked on the 2011 Performance Ranking of Scientific Papers for World Universities, prepared 
by the Higher Education Evaluation & Accreditation Council ofTaiwan, and the SiR World Report 2012 
Global Ranking, by SCIMago Research Group using the Scopus research publications data base. 


Table 4. Australian Universities ranked by Taiwan HEEAC, 201 1, and the SCIMago Research 
Group, 2012 


Performance Ranking of Scientific 
Papers for World Universities, 201 1 

SIR World Report 2012 Global Ranking 

The University of Melbourne (45) 

The University of Sydney (59) 

The University of Sydney (70) 

The University of Melbourne (62) 

University of Queensland (76) 

University of Queensland (74) 

Monash University (1 1 3) 

The University of New South Wales (83) 

The University of New South Wales (1 28) 

Monash University (1 1 2) 

The Australian National University (1 74) 

The Australian National University (1 85) 

The University of Western Australia (190) 

The University of Western Australia (227) 

Walter & Eliza Hall Institute of Medical Research (203) 

The University of Adelaide (289) 

The University of Adelaide (243) 

Queensland University of Technology (499) 

Macquarie University (391) 

Curtin University ofTechnology (509) 

James Cook University (439) 

The University of Wollongong (547) 

University ofTasmania (448) 

The University of Newcastle (568) 

The University of Newcastle (452) 

Macquarie University (569) 

See Appendix 1 for an outline of the HEEAC and SIR rankings. 

Griffith University (576) 

University ofTechnology Sydney (679) 
University ofTasmania (680) 

RMIT University (708) 

La Trobe University (71 1) 

Deakin University (714) 

University of South Australia (734) 
Flinders University (763) 

James Cook University (766) 

University of Western Sydney (983) 


Table 5 shows the position of Australian universities on the National Taiwan University Rankings for 
scientific papers in 201 1 and 2012. 
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Table 5. Australian universities on the National Taiwan University Rankings for scientific papers, 
2011 &2012 


1 University 

2011 

2012 1 

University of Melbourne 

41 

35 

University of Sydney 

66 

61 

University of Queensland 

79 

72 

Monash University 

122 

112 

University of New South Wales 

127 

1 14 

Australian National university 

160 

172 

University of Western Australia 

162 

179 

Walter & Eliza Flail Institute 

219 

207 

University of Adelaide 

236 

252 

Macquarie University 

373 

455 

James Cook University 

400 

465 

University of Newcastle 

487 

465 

University ofTasmania 

392 

477 

University of Wollongong 

464 

NR 

Flinders University 

486 

NR 


From these various sources it is evident that while there are changes in the position of some 
universities on different measures there is a common group of around 20 Australian universities 
positioned in the top 1,000 on several of the measures. 

Some disaggregation of rankings is possible for particular fields and subjects, although field 
classifications vary by ranking agency. Table 6 shows sub-institutional rankings of the ARWU. For 
instance, in Life & Agricultural Sciences, UWA, Melbourne and UQ are ranked respectively 26, 42, 
and 45, and all Go8 universities are ranked within the top 150 in the world. ANU is ranked 30 in 
Physics and Melbourne 35 in Clinical Medicine and Pharmacy. 

Table 6. Australia's Go8 Universities - ARWU Rankings for Fields and Subjects, 2012 


Field 


Go8 University 

Natural 
Sciences and 
Mathematics 

Engineering 
Technology 
and Computer 
Sciences 

Life and 
Agricultural 
Sciences 

Clinical 
Medicine and 
Pharmacy 

Social Sciences 

UMelbourne 

10M50 

51-75 

42 

35 

101-150 

ANU 

37 

101-150 

51-75 


76-100 

UQ 

151-200 

76-100 

45 

51-75 

101-150 

U Sydney 

101-150 

51-75 

76-100 

101-150 

101-150 

UWA 

101-150 


26 

51-75 


Monash 

101-150 

51-75 

101-150 

1 5 1 -200 

151-200 

UNSW 

151-200 

51-75 

101-150 

101-150 

101-150 

UAdelaide 



101-150 

151-200 
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Subject 



Go8 University 

Mathematics 

Physics 

Chemistry 

Computer 

Science 

Economics/ 

Business 

UMelbourne 

101-150 

76-100 

151-200 

101-150 

76-100 

ANU 

151-200 

30 


101-150 

151-200 

UQ 



151-200 


101-150 

U Sydney 

101-150 

101-150 

76-100 


101-150 

UWA 



76-100 



Monash 



51-75 

76-100 

151-200 

UNSW 

76-100 


101-150 

101-150 

101-150 

UAdelaide 


Several other field-specific rankings reinforce the impressions of the major institutional rankings 
but also reveal greater degrees of variation among institutions. In the 201 2 Financial Times Global 
MBA Rankings, The Australian School of Business (AGSM) at the University of New South Wales was 
placed 37 and Melbourne Business School 54 in the top 1 00 institutions. In the Economist ranking 
of full-time MBA programs the University of Queensland Business School was ranked 27 and the 
University of Melbourne Business School 38. Two other Australian universities were ranked in the 
Economist top 1 00 full-time MBAs: Curtin Graduate School of Management at 46 and Macguarie 
Graduate School of Management at 61 . The Economist invited 1 35 business schools to respond to 
a two-staged survey, one involved responses of students and alumni to a guestionnaire (20%) and 
the other reguired the schools themselves to respond to guestions about career opportunities for 
graduates, the educational experience of students, increases in graduate salaries and potential for 
graduates to network (80%). 

In 201 2, the IDEAS grouped ranked the Business/Economics departments of 5,858 institutions 
worldwide, drawing on several data bases hosted at the RePEc service of the federal Reserve Bank 
of St Louis. The rankings are derived from bibliometric data collected by RePEc, citations analysis 
performed by CitEc and popularity data compiled by LogEc. Table 7 shows the placement of 
faculties and Schools of Australian universities. 


Table 7. World Ranking of Australian Business/Economics Departments, 2012 


1 Department/Institution 

Rank I 

Faculty of Business and Economics, Monash University 

25 

Crawford School, Australian National University 

66 

Faculty of Business and Economics, University of Melbourne 

75 

Australian School of Business, University of New South Wales 

89 

School of Economics, University of Queensland 

97 

Department of Economics, Faculty of Business and Economics, Monash University 

98 

School of Economics, Australian School of Business, University of New South Wales 

107 

College of Business and Economics, Australian National University 

121 

Research School of Economics, Australian National University 

123 

Business School, University of Technology Sydney 

141 

Department of Economics, Faculty of Business and Economics, University of Melbourne 

144 


In Philosophy, ANU sits within the top 20 on the Philosophy Gourmet rankings (see Table 8). 
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Table 8. Philosophy Gourmet Rankings of Faculties in the English-Speaking world: Top 20, 201 2 


School 

Rank 

New York University 1 

Oxford University 

2 

Rutgers University, New Brunswick 

3 

Princeton University 

4 

University of Michigan, Ann Arbor 

5 

Harvard University 

6 

University of Pittsburgh 

6 

Massachusetts Institute ofTechnology 

8 

Yale University 

8 

Stanford University 

10 

University of North Carolina, Chapel Hill 

10 

Columbia University 

12 

University of California, Los Angeles 

12 

University of Southern California 

12 

Australian National University 

15 

City University of New York Graduate Center 

15 

Cornell University 

15 

University of Arizona 

15 

University of California, Berkeley 

15 

University of Toronto 

15 


Source: Philosophy Gourmet Rankings of Faculties in the English-Speaking world, 201 2 

Table 9 shows the position of Australian universities in the broad fields of the National Taiwan 
University's 201 2 rankings for scientific papers. Melbourne ranks at the top of Australian 
universities in the fields of Medicine and Life Sciences, whereas UQ leads in Agriculture and 
Engineering, Sydney in Social Sciences, and ANU in Natural Sciences. The standings reflect scale of 
the guality outputs counted. 


Table 9. Australian Universities on Broad Fields Rankings, National Taiwan University Rankings 
for Scientific Papers, 201 2 


University 

Agriculture 

Medicine 

Engineering 

Life Sciences 

Social Sciences 

Natural 

Sciences 

Melbourne 

41 

27 

87 

33 

50 

97 

Sydney 

52 

45 

106 

81 

40 


Queensland 

12 

100 

62 

51 

49 

118 

Monash 

138 

90 

96 

90 

75 

165 

UNSW 

117 

no 

72 

108 

62 

130 

ANU 

48 


236 

232 

85 

90 

UWA 

55 

153 


162 

140 

256 

Adelaide 

64 

190 


247 

296 


Macquarie 

137 






James Cook 

74 






Newcastle 


276 



201 


Tasmania 

116 






Wollongong 

Murdoch 



298 


276 


Griffith 

267 




231 


Deakin 





186 


QUT 





215 


LaTrobe 





223 


UTS 





269 


UniSA 





270 


Curtin 





278 



Source: National Taiwan University 
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To the extent that the rankings, when taken together, paint a generai iandscape over which 
universities spread, they reinforce domestic indicators of reiative strengths in academic capacity, 
performance and reputation. Those domestic indicators (see Appendix 2) inciude input indicators 
such as institutionai asset strength and annuai operating revenue, entry standards of students, 
progression and compietion rates of students, academic staff guaiifications, and competitiveiy- 
won income for research from government and industry, aiong with output indicators inciuding 
graduate empioyment or further iearning, graduate income, research pubiications, citations and 
measures of esteem, and evaiuations of research guaiity such as the Exceiience for Research in 
Austraiia (ERA) ratings (see Eigure 1). 

Figure 1 . ERA 201 0 Research quality assessment ratings of 4 ('above world standard') 
and 5 ('well above world standard') 


The University of Melbourne 
The University of Queensland 
The University of Sydney 
The University of New South Wales 
The Australian National University 
Monash University 
The University of Adelaide 
The University of Western Australia 
Macquarie University 
The University of Wollongong 
Queensland University of Technology 
Griffith University 
The University of Newcastle 
The University ofTasmania 
Deakin University 
LaTrobe University 
James Cook University 
Murdoch University 
The University of South Australia 
Curtin University of Technology 
The University of Technology, Sydney 
RMIT University 
Flinders University 
The University ofWestern Sydney 
Charles Darwin University 
Swinburne University of Technology 
Charles Sturt University 
The University of New England 
Victoria University 
The University of Canberra 
Southern Cross University 
Edith Cowan University 
The Australian Catholic University 
Bond University 
The University of Southern Queensland 
Central Queensland University 
The University of Ballarat 
The University of the Sunshine Coast 
Melbourne College of Divinity 
The University of Notre Dame Australia 
Batchelor Institute of Indigenous Tertiary Education 

0 15 30 45 60 75 90 105 

Figure 1 shows the concentration of research strengths (ERA ratings of 4 and 5) among Australian 
universities. There is a range within the Go8 in terms of the comprehensiveness of research 
strength, a wide gap between the Go8 and the next ranked universities, and a tail of universities 
with few if any strong areas of research performance. The concentration is even more apparent 
when ratings of 5 are compared. Whereas the Go8 universities had multiple fields rated 5 (e.g. 1 2 
in the case of Melbourne and 1 1 for ANU) there were only nine other universities which gained 
a 5 rating and for each of them in only one field. Even so, the ERA results confirm the impression 
given by the various world rankings that there are some 20 or so Australian universities where 
high quality research is seen to be performed to some extent, and that is important for Australia in 
serving diverse needs. What differentiates the Go8 is the scale of high quality research performed. 
Go8 universities are more comprehensive in research, including health and medical research, and 
have higher numbers and proportions of research students, than non-Go8 universities. 

As can be seen from comparing tables 1 0 and 1 1 , over the period 2006-201 0 the Go8 universities 
have produced ten times the volume of journal articles co-authored with international institutions 
that the ATN universities produced. Additionally, Go8 universities have greater intensity of co- 
authorship with the most highly ranked overseas universities. 
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Table 1 0. Internationally co-authored papers, Go8 universities, 2006-201 0 


1 Institution 

Co-authored articles I 

Harvard University 

1480 

University of Oxford 

1316 

University College London 

1066 

University of Cambridge 

1057 

National University of Singapore 

1035 

University of Toronto 

920 

Imperial College London 

878 

University of Auckland 

846 

University of Otago 

729 

University of Edinburgh 

687 

University of Washington 

640 

University of Copenhagen 

639 

University of California at Los Angeles 

590 

University of British Columbia 

587 

Johns Hopkins University 

586 

King's College London 

579 

Stanford University 

556 

University of Bristol 

523 

University of California at Berkeley 

516 

University of Manchester 

513 

University ofTokyo 

507 

University of Glasgow 

477 

University of Leeds 

474 

University of Michigan 

472 

Chinese Academy of Sciences 

469 

University of California at Davis 

465 

University of Hong Kong 

457 

University of Hawaii 

454 

University of Pennsylvania 

448 

Columbia University 

447 

NanyangTechnological University 

442 

University of Alberta 

437 

Karolinska Institutet 

435 

CNRS 

427 

University of California at San Diego 

406 

Hong Kong Polytechnic University 

400 

University of Birmingham 

400 

University of Southampton 

398 

University of Wisconsin 

396 

McGill University 

392 

University of California at San Francisco 

388 

California Institute ofTechnology 

387 

University of Minnesota 

385 

University of Pittsburgh 

380 

Peking University 

375 

Cardiff University 

375 

Lund University 

372 

University of Aarhus 

371 

University of Zurich 

361 


SciVal Spotlight Co-Citation Analysis, 201 2 Elsevier 
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Table 1 1 . Internationally co-authored papers, ATN universities, 2006-201 0 


1 Institution 

Co-authored articles I 

Hong Kong Polytechnic University 

162 

National University of Singapore 

159 

Chinese Academy of Sciences 

1 14 

University of Hong Kong 

no 

University of Manchester 

107 

City University of Hong Kong 

105 

University of Cambridge 

88 

University of Otago 

84 

Sun Yat-Sen University 

78 

Peking University 

77 

NanyangTechnological University 

77 

University ofToronto 

76 

University of Auckland 

74 

Harvard University 

74 

Eindhoven University ofTechnology 

73 

University of Calgary 

69 

McGill University 

64 

Harbin Institute ofTechnology 

63 

University of Nottingham 

63 

Zhejiang University 

61 

CAS - Guangzhou Institute of Geochemistry 

61 

University of California at Los Angeles 

61 

CAS - Institute of Geology and Geophysics 

60 

University of Bristol 

60 

University of Oxford 

60 

University College London 

59 

Tsinghua University 

57 

Karlsruhe Institute ofTechnology KIT 

57 

Chinese University of Hong Kong 

56 

Nanchang University 

55 

University of Copenhagen 

54 

University of Alberta 

52 

University of Michigan 

52 

University of Sheffield 

51 

University of Pennsylvania 

51 

University of Washington 

51 

Graduate University of Chinese Academy of Sciences 

49 

World Health Organization 

49 

University of Newcastle upon Tyne 

48 

University of Cape Town 

47 

ETH Zurich 

47 

Erasmus University Rotterdam 

47 

Massey University 

46 

King's College London 

46 

University of British Columbia 

46 

Pennsylvania State University 

46 

Imperial College London 

45 

University of Waterloo 

45 

Shanghai Jiaotong University 

44 


SciVal Spotlight Co-Citation Analysis, 201 2 Elsevier 


Go8 BACKGROUNDER 30 | WORLD UNiVERSiTY RANKINGS: AMBiGUOUS SiGNALS 


PAGE 17 





























The position of Australia's leading universities on world rankings may also be construed as 
validating judgements of relative international standing, or indications of comparable peers for 
purposes of collaboration or partnering. Table 12 shows the placement of Australia's leading 
universities on the ARWU rankings alongside the top research universities of North America, i.e. 
members of the Association of American Universities (AAU). Clearly Australia's public universities 
do not match the strength of the US Ivy League but they position reasonably well against the 
land-grant public four-year colleges. 


Table 12. ARWU Rankings 2012: Association of American Universities (AAU) and Go8 


Harvard University 


Stanford University 


Massachusetts Institute ofTechnology 

University of California, Berkeley 

California Institute ofTechnology 

Princeton University 

Columbia University 


Purdue University 
University of Pittsburgh 


2 |The University of Melbourne 

3 


Rutgers, The State University of New Jersey 
McGill University 


The Australian National University 


Brown University 


University of Chicago 


The Ohio State University 


Yale University 


University of California, Los Angeles 


University of Florida 


University of Arizona 


Cornell University 


Indiana University Bloomington 


University of Pennsylvania 


University of Rochester 


The Pennsylvania State University 

14 

The University of Queensland 

90 

University of California, San Diego 

15 

Rice University 

■ 91 

University of Washington 

16 

The University of Sydney 

93 

Johns Hopkins University 

17 

Texas A&M University 

■ 93 

University of Wisconsin-Madison 

19 

The University of Western Australia 

96 


University of Michigan, Ann Arbor 

University of Illinois at Urbana-Champaign 
New York University 


Michigan State University 
I Case Western Reserve University 


University of Virginia 


University of Toronto 


Emory University 


University of Minnesota, Twin Cities 


Northwestern University 


Washington University in St Louis 

University of Colorado at Boulder 

University of California, Santa Barbara 

The University ofTexas at Austin 

Duke University 

University of Maryland, College Park 
University of North Carolina at Chapel Hil 
University of California, Irvine 


University of Iowa 


30 Monash University 

31 The University of New South Wales 
33 


Iowa State University 


Stony Brook University (the State University of 
New York at Stony Brook) 


University of Southern California 
University of California, Davis 


Vanderbilt University 


Carnegie Mellon University 


University of Nebraska-Lincoln 


The University of Adelaide 


University of Kansas 
University of Missouri 
University of Oregon 
University of Buffalo, The State University of 

New York 

Brandeis University 
Tulane University 
Syracuse University 


Table 1 3 compares ARWU rankings for the Go8 and Canada's U1 5 group of research-intensive 
universities. This general fit of the leading universities of two major second world powers with 
dominant English colonial backgrounds is not unexpected. 
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Table 13. ARWU Rankings 2012: Canada U15 and Go8 


University ofToronto 


The University of New South Wales 

142 

University of British Columbia 

University of Waterloo 

198 

The University of Melbourne 

57 

University of Calgary 

204 

McGill University 

63 

University of Western Ontario 

211 

The Australian National University 

64 

The University of Adelaide 

221 

McMaster University 

92 

Queen's University 

237 

The University of Queensland 

90 

University of Ottawa 

249 

The University of Sydney 

93 

Dalhousie University 

253 

The University of Western Australia 

96 

University of Saskatchewan 

262 

University of Alberta 

108 

Laval University 

265 

Universite de Montreal 

132 

University of Manitoba 

286 

Monash University 

138 




Australia's research-intensive universities also compare well with the UK's Russell Group, as shown 
in Table 14. While Australia cannot quite replicate the special character of Oxford, Cambridge, UCL 
and Imperial, there is not a great distance between the next group of strong research universities, 
what Warwick Vice-Chancellor NigelThrift refers to as the'fertile crescent'of universities "without 
the natural advantages of the big four which has been one of the true success stories of the British 
higher education sector in the years since the 1 960s" (Thrift, 2009). 


Table 14. ARWU Rankings 2012: Russell Group (UK) and Go8 


The University of Cambridge 

5 

The University of Birmingham 

110 

The University of Oxford 

10 

The University of Sheffield 

106 

University College London 

m 

London School of Econ. & Pol. Sci. 

120 

Imperial College 

The University of Liverpool 

126 

The University of Manchester 

40 

Monash University 

138 

University of Edinburgh 

51 

The University of New South Wales 

142 

The University of Melbourne 

57 

The University of Leeds 

153 

The Australian National University 

64 

The University of Glasgow 

159 

King's College London 

68 

Cardiff University 

160 

The University of Bristol 

70 

The University of Southampton 

164 

The University of Nottingham 

86 

The University of Warwick 

177 

The University of Queensland 

90 

Newcastle University 

216 

The University of Sydney 

93 

The University of Adelaide 

221 

The University of Western Australia 

96 

Queen's University Belfast 

327 
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The up-sides of being well ranked 
and reasonably well ranked 

Being placed among the top universities in the world on a range of rankings has reputational and 
strategic positioning advantages for individual universities and their home countries, and status 
benefits for the students and alumni of institutions. While the benefits may be intangible - they 
may motivate a university's internal community and give confidence to its external community 
- they can also have material effects on graduate employment and earnings, and institutional 
revenue from student recruitment, industry funding, philanthropic giving and government 
investment, as well as opening up opportunities for networking and partnering. 

High positional rankings of universities, even if they are based only on a limited set of research 
metrics, signal capability to the world - research capacity, facilities, expertise and linkages. To the 
extent that they attract talent (students and staff) and investment (R&D) they can be powerful 
drivers of expanding capability. 

Interestingly, none of the major world rankings provide direct measures of teaching and learning 
quality, even though they are referenced by prospective international students as a guide to 
relative educational quality and prestige (IDP forthcoming). Largely because they reflect, or 
purport to reflect, core values of the academy, world university rankings indirectly underpin 
the reputation of national qualifications and the international education industry. The rankings 
function, however obliquely, to confirm claims to international standing and indicate that 
graduates can take their place in work and further learning anywhere in the world. 

However, it is this usage of the rankings that is the most problematic for two reasons: first, 
it involves inferences about educational quality that cannot be drawn necessarily from the 
information provided but, rather, are derived from assumed relations among academic functions 
of the traditional university of the period of elite participation; and second, it fails to distinguish 
between different orientations to teaching and learning in different institutions and the 
corresponding differences in attributes of graduates. 

Horta (2009) distinguishes between 'prominent national universities'- those with the greatest 
capacity or strongest performance with regard to other institutions in a country - and 'prominent 
global universities'- those that are world leaders in terms of their capacity and performance. This 
is to acknowledge that a dominant national position may fall well short of global prominence, that 
"success at the top of the system is determined in the international league of champions not the 
national league" (Teichler, 2006). What matters most is how well the best national universities of a 
particular orientation compare with the world's best with a similar orientation - for that sets the 
peak achievement standards for the system. It is not possible for a nation to have a 'world class 
higher education system' without some 'world class universities'. That status is not referenced to the 
performance mean of the world's universities ranging from Azerbaijan to Zimbabwe but rather to 
those sets of institutions which define the class of world leaders in higher education and research. 

Importantly, there are multiple orientations among universities and there should be therefore 
multiple sets of criteria for evaluating what constitutes 'leading'and who measures up. 

The rankings also imply broader institutional capacities (e.g. good university governance & 
management) even though the efforts being captured by the rankings metrics are chiefly those 
of individual researchers and groups of researchers. The point of good university governance is to 
create the room for academic achievement - through cultural values, financial incentives, physical 
facilities, clear focus on strategic directions and responsiveness to changing circumstances. To 
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some extent the rankings can reflect, and also influence, corporate university investment priorities 
in facilities and talent, such as key centres in fields of knowledge. Nevertheless, one university's 
position on the rankings cannot be managed simply by its own efforts. This is because of constant 
change in the operating environment, the efforts of others to improve their positions, and 
aspects of the rankings tools themselves - both changes in methods of counting and inherent 
methodological deficiencies - which render year-on-year results volatile and unpredictable. 


The danger of complacency 

Australia may be seen to have advanced from 14 universities in the top 500 in 2005 to 19 in 
201 2 on the research-focused ARWU scale - an apparent rise for 5 universities, although some 
institutions have slipped in their positions over that period. However, over the same period, China 
has moved from 8 to 28 universities in the top 500 - an apparent rise of 20 universities. These are 
apparent rises because there have also been changes to the calculation of some of the metrics 
over the period to the relative advantage of particular institutions. 

Importantly, the indicators used in all of the major university rankings are necessarily lagged. In 
the case of the ARWU rankings, the Highly Cited indicator is based on cited research on the Web of 
Science database for the period 1981-2008. Yet since 2002, India and South Korea have overtaken 
Australia in research output and China has outstripped Australia's output four times. Concurrently, 
all developed economies, other than the US, UK and Finland, have increased their number of 
citations per paper at a faster rate than Australia (Barlow, 201 Ij.Thus we can expect to see Asian 
universities rising further in the rankings as more current indicators of research performance are 
factored into the rankings metrics. 

It is not that Australia has a large lead on the upcoming countries. Australia is ranked 1 0 among 40 
countries in terms of publications in the top guartile journals per million dollars GDP (OECD, 201 2). 
Australia's performance falls below the OECD mean on the scale of international co-authorship as 
a proportion of total scientific articles produced, and is ranked 26 out of 46 countries. Ireland is 
ranked 13, New Zealand 14, the Netherlands 1 1, Switzerland 4 and Indonesia 3 (OECD, 2012). 

Australia's apparent rise at the top end of the ARWU scale has been influenced in part by recently 
awarded Nobel laureates from UWA (2005) and ANU (201 1 ). Where Nobel winners are on staff they 
contribute one fifth of the SJT rank score. Robin Warren and Barry Marshall of UWA made their 
discovery of the Helicobacter pylori bacterium in 1982. Brian Schmidt (ANU) produced his paper 
(with Saul Perlmutter and Adam Reiss) on the expanding universe in 1 998. A tapered weight of 
1 0 per cent is given for former students and staff of a university who are awarded a Nobel Prize. 
Peter Doherty, an alumnus of UQ, shared a Nobel Prize in Physiology with Swiss Professor Rolf 
Zinkernagel for work on the immune system he carried out at ANU between 1 973 and 1 975. 

A comparison of raw rankings across countries can give misleading impressions of performance 
relative to scale and capacity. Eigure 2 compares the number of top ranked universities on the 
201 2 ARWU rankings by country per million population and per USD 1,000 of Gross Domestic 
Product per capita. The US and Ghina stand out as achieving a high apparent outcome relative 
to economic capacity but a lower outcome per head of population. Australia and the UK have 
commensurate outcomes per head of population but the UK achieves more than Australia relative 
to economic capacity. 
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Figure 2. Number of universities in the ARWU top 200 per million of population and US$1000 
of GDP per capita, 201 2, by selected country 


2 -i 


Top 200 universities per 
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Top 200 universities per 
US$1000 of GDP per capita 


1 - 


0.8 - 
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Figure 3 compares ARWU rankings per USD 1 ,000 million invested in tertiary education. With 
regard to 'return on investment' Australia appears to perform much better than some countries 
(including the US), but not nearly as well as Israel, Switzerland, the UK and the Netherlands. This 
suggests that the latter focus a higher proportion of their tertiary education funding on university 
research than Australia which gives priority to student participation. 


Figure 3. Number of universities in the ARWU top 200, by selected country, per billion dollars 
(US) spent on tertiary education institutions 

1.4 1 
1.2 - 
1 - 


0.8 - 



Australia US UK Germany Switzerland Netherlands Canada Israel China South Korea 


Note; ARWU rankings are for 2012; expenditure data are from OECD Education at a Glance 201 1 (2008 data). Expenditure includes both 
public and private sources, except for Switzerland, for which no private expenditure data are available. Data for expenditure on tertiary 
education institutions was not available for China. 
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Figure 4 normalises the 2012 ARWU rankings for national population size and wealth (GDP per 
capita) referenced to Australia. The graph shows the number of universities in the top 200 by 
country, plus what the number'would be'assuming the same relationship between ARWU 
performance and national population and GDP, respectively, if each country had the same 
population and GDP per capita as Australia. In general, Australia punches above its demographic 
weight, getting more universities in the top 200 per million of population than several of the 
biggest countries with strong universities (US, UK, Germany). Several countries smaller than 
Australia do better on a per capita basis (the Netherlands, Denmark, Sweden and Israel). Australia's 
performance is not especially distinguished once our high GDP per capita is controlled for. The 
US performs vastly better, while the UK and Germany are also decidedly ahead of Australia. Other 
small, prosperous countries are comparable to Australia, while Ganada is slightly ahead. Most 
interestingly, Ghina is well ahead of Australia on this measure, and out-performs other advanced 
countries too - with the sole exception of the US. 

Figure 4. Number of universities in the ARWU top 200, by selected country, 201 2: actual, 
normalised to Australian population and Australian GDP per capita 
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Population data are taken from the Population Reference Bureau's World Population Reference Sheet for 201 2 (i.e. 201 2 estimates). GDP 
per capita data are 201 1 figures from the World Bank, in current US dollars. 


Figure 5 shows relative ranking outcomes on theTFIEThomson Reuters rankings. This ranking 
improves the performance of the UK, European and Australian systems relative to the US and Ghina. 
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Figure 5. Number of universities in the THE Thomson Reuters top 200, by selected country, 
201 1-1 2: actual, normalised to Australian population and Australian GDP per capita 



Figure 6 shows the QS top 200 in 201 1 by country normalised to Australia's population and GDP 
per capita. Interestingly, although Australia has a slightly better actual performance on the QS 
ranking than on the ARWU and THE-TR rankings, Australia's relative performance is weaker than on 
those other measures. 

Figure 6. Number of universities in the QS top 200, by selected country, 201 1 : actual, 
normalised to Australian population and Australian GDP per capita 
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An evident trend in recent THE and QS world rankings is the appearance of new universities, 
notably those from countries not previously represented in the rankings. This trend reflects in 
part the influence of demographic factors on the reputational surveys which count towards 
institutions' scores on theTHE and QS measures. Importantly, many Asian institutions are recipients 
of increased government investment in higher education and university research, substantially 
so in China. In contrast, several former leading universities in the US and the UK are experiencing 
the effects of disinvestment as government funding falls and annual returns from university funds 
under investment deteriorate. The shift of relative input strength in favour of Asian universities will 
show through in the rankings that make use of input measures. 

Efforts by the university rankers to improve their methods, including efforts to reduce country 
bias in reputational surveys, and to expand the country markets for their rankings, suggest that 
the earlier dominance of universities of the developed economies of North America, Britain and 
Europe will decline over time. This tendency will be more evident in those rankings, such asTHE 
and QS, that put less weight on the scale of high guality research. 

A comparison of the position of top ranked Asian universities on theTHE-Thomson Reuters 
ranking shows a relatively stable group of leading Asian institutions over the last three years, with 
Singapore's two leading institutions rising (see Table 15). 


Table 15. Asian universities in the top 200 of theTHE World University Rankings, 2010, 201 1 & 2012 


1 Institution 

2010 

201 1 

2012 1 

Tokyo University 

26 

30 

27 

National University of Singapore 

34 

40 

29 

University of Hong Kong 

21 

34 

35 

Peking University 

37 

49 

46 

Tsingua University 

58 

71 

52 

POSTECH 

28 

53 

50 

Kyoto University 

57 

52 

54 

Hong Kong University of Science &Technology 

41 

62 

65 

KAIST 

79 

94 

68 

Nanyang Institute of Science &Technology 

174 

169 

86 

Chinese University of Hong Kong 

NR 

151 

124 

Tokyo Institute of Science & Technology 

112 

108 

126 

National Taiwan University 

115 

154 

134 

Tohuku University 

132 

120 

137 

Osaka University 

130 

119 

147 

City University of Hong Kong 

NR 

193 

182 

University of Science &Technology China 

49 

192 

NR 


The more dynamic developments are within the next tier of Asian universities. Those in the top 
400 in theTHE 2012-13 rankings include, in order: Eudan University; City University of Hong 
Kong;Yonsei University; Hong Kong Polytechnic University; IIT Bombay; Nanjing University; 

Tokyo Metropolitan University; Kyushu University; Renmin University of China; Shanghai Jiao 
Tong University; Tokyo Medical and Dental University; National Cheng Kung University; Hokkaido 
University; Hong Kong Baptist University; National Sun Yat-Sen University; Sun Yat-Sen University; 
University of Tsukuba; Zhejiang University; National Central University, Taiwan; IIT Roorkee; Keio 
University; King Mongut's University ofTechnology,Thonburi; Waseda University; National Taiwan 
University of Science and Technology. 

Australian universities may appear to be improving against the backdrop of decline in the US and 
the UK, whereas they may well be slipping behind the emergent Asian leaders. 
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Methodological concerns 

Since the appearance of the Shanghai Jiao Tong (ARWU) rankings in 2003 there has been a 
proliferation of critiques of rankings methodologies (see for instance: van Raan, 2007; Salmi & 
Sayoran, 2007; HEFCE, 2008; Billaut et al, 2010), and various efforts on the part of the rankers 
themselves to respond to the criticisms and improve their methods (Longden, 201 1 ). 

A core criticism is that the rankings are largely shaped by the limited availability of comparable 
data, rather than by any concept of desirable indicators (Teichler, 201 1). This criticism is not entirely 
fair given the considerable effort involved by the academy and publishing houses to develop 
serious measures of leading research. The validity problem - the lack of alignment between 
the evaluative criteria and the measures used - is most troublesome for comparisons relating 
to teaching and learning. Nor can such a criticism be levelled only at university rankings. An 
alternative to rankings, the U-Map project, also suffers from the same limitation (see Box 1 ). For 
example, U-Map has no indicators of quality of teaching and learning; it uses start-ups and patents 
as proxies for knowledge exchange, not because they inform about knowledge exchange but 
because there are data available. Similarly, international orientation does not include any measure 
of curriculum internationalisation. The appropriate response to the data-related criticism is to limit 
the application of research-based rankings to research-intensive universities, and to construct 
other measures for teaching-intensive universities and other higher education institutions. 


Box 1. U-Map dimension and indicators 

1 Teaching and learning profile 

Student profile 

Research involvement 1 

• Degree level focus 

• Mature students 

• Peer reviewed publications 

• Range of subjects 

• Part-time students 

• Doctorate production 

• Orientation of degrees 

• Distance learning students 

• Expenditure on research 

• Expenditure on teaching 

• Size of student body 


• Involvement in knowledge 
exchange 

• International orientation 

• Regional engagement 

• Start-up firms 

• Foreign degree seeking 

• Graduates working in the 

• Patent applications filed 

students 

region 

• Cultural activities 

• Income from knowledge 
exchange activities 

• Incoming students from 
international exchange 
programmes 

• Students sent out in 

• First year bachelor students 
from the region 

• Importance of local/regional 
income sources 


international exchange 
programmes 



• International academic staff 



• The importance of 
international sources of 



income in the overall budget 
of the institution 


Source: van Vught et at, 201 0 


A second basic criticism of rankings is that they are biased by favouring English-speaking, 
research-intensive institutions with strengths in natural sciences. One could add to the bias in that 
the rankings tend to favour older, well-resourced prestigious universities. However, the bias does 
not derive from any prior selecting of such institutions but from the choice of output indicators, 
such as highly cited journal articles, which some institutions produce more than others. The 
ARWU, for instance, gives weight to articles published in Nature and Science to the disadvantage of 
disciplines which publish elsewhere. 
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Differences in the communication channels of disciplines are generally not respected by the main 
world university rankings. Yet publishing practices are very diverse (Sparks, 2005). Whereas journals 
are important in medical and biological sciences, e-prints and technical reports are important in 
the physical sciences and engineering, as are peer-reviewed conference proceedings. Professional 
magazines are important in fields like accounting, business, commerce, law, management, and 
public health. Monographs, book chapters and other books are important for languages and 
area studies. Some specific disciplines and cross-disciplinary studies, rely more than others on 
publishing via artefacts, exhibitions, patent applications, performances, and software. Some fields of 
engineering, for instance, make substantial use of conference proceedings and other outlets which 
are determined according to the relevance of the research to national and international industries 
(Kaufman & Kasztier, 2005). In the field of Law, except for areas of international law, research and 
scholarship is typically context-dependent or jurisdiction-specific and referenced to legislative 
issues in local parliaments and case law developments in local courts. In the Australian context, 
scholarly publications in Law include the Federal Law Review, Sydney Law review, Melbourne University 
Law Review, Australian Business Law Review, Australian Journal of Family Law. While these may be well 
regarded locally their articles are not necessarily cited widely in the international literature. 

Additionally, the research output measures of the world university rankings, especially citation 
impact, favour discipline-based over cross-disciplinary research, and basic over applied research. 

Conseguently, the data sources for compiling some of the indicators used to derive world 
university rankings do not capture important outputs of different universities in different fields. 

On the one hand, the rankings tend to under-rate the significance of research for local uses and 
under-estimate the strengths of those institutions whose outputs are not highly visible on the 
indicators selected by rankers. On the other hand, in the changing global context, some of the 
data collection biases of the current suite of world university rankings may become less potent as 
non-Anglo researchers increase their co-authored publications, as non-English publications gain 
international traction, and as open source publishing becomes more common - to the extent that 
it is not limited by counter-terrorism efforts. 

A third criticism is that the weights assigned to different elements within the rankings metrics 
are arbitrary. Clearly that is so; none of the rankings has derived its weights from any conceptual 
framework or statistical procedure. Different institutions could well be ranked higher or lower if 
the weightings were varied. However, the problem is that there is no consensus on any 'best fit' 
set of weights. For instance, the AWRU gives 30% to Nobel Laureates. Perhaps laureates should 
not be included at all, as their inclusion reflects a neo-Confucian view of leadership in university 
research and education, and give higher weights to other indicators; but if they are to be included 
as a proxy for a university culture that enables exceptional performance at the peak of intellectual 
attainment, they must be accorded significance. 

Nevertheless, there are basic statistical problems with the way world university rankings are 
constructed. It is not only that weights are assigned arbitrarily to different indicators. There is also 
no correction for redundant measures - the interactions among the variables. Additionally, while 
some of the rankings normalise the data for some indicators, there is no uniform or systematic 
standardisation of the distribution of scores. Hence, indicators with a wide spread of scores (e.g. 

1 0-90) will have a greater influence on the final score than those indicators with a narrower spread 
(e.g. 40-65) - before any weights are applied (HEFCE, 2008). 

A fourth (and related) criticism relates to the subjectivity of several of the indicators in some of the 
rankings, and their proneness to manipulation. The THE-Thomson-Reuters rankings give 15% and 
1 8% respectively to teaching reputation and research reputation, a total of one-third. QS gives 40% 
to academic peer review and 1 0% to recruiter review, a total of one half The reputation surveys 
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have experienced problems of low response rates with potential non-response bias, country 
sample bias and rater bias. Importantly, the processes by which the survey data are collected 
and reported are opaque, in terms of populations surveyed, questions asked, response rates and 
classification of responses. Such lack of transparency for such a high level of ranking dependency 
on the subjective measures undermines confidence in the published rankings, even if the rankers 
seek to reduce transparency in order to minimise the risk of institutional gaming. 

One could argue that peer review is a longstanding basis for determining and validating quality 
in academic matters, and that for all its faults, like democracy, it is hard to find a better model. 
Nevertheless, peer review is not without its faults and much depends on which peers are involved 
in making judgements (Aguillo et al, 201 0; Bookstein et al, 201 0) and the range of criteria they apply 
(Scott, 2006). Additionally, there is the risk of gaming, such as through exercises to increase the number 
and preferences of peer reports. Other indicators, such as student staff ratios and the inclusion of 
publishing outputs by researchers of affiliated institutions, are also prone to institutional manipulation. 

A fifth line of criticism is that there is no theorised balance between input, process and output 
indicators and no attempted relationship between such indicators, such as to identity value added. 
THE-Thomson-Reuters and OS rankings mix input and output indicators without an underlying 
rationale. Using only output indicators could be unfair (if fairness should be a consideration in 
assessing excellence) when comparisons are not made on a like-to-like basis in terms of scale. 
Significantly, inputs matter in higher education no less than in other systems where they function 
as means to control outputs. Indeed to the extent that learning is co-produced, student and 
teaching staff inputs matter substantively. Additionally, learning quality is influenced by the 
'customer-input-technology' of bright students asking probing questions just as much as by 
quality teachers. Research output is dependent on the inputs of good researchers with adequate 
resources to do their work. Seeking measures of value added in discovery and learning assumes 
a purpose beyond simply ranking, including but not limited to improvement, and it would add 
enormously to complexity and data gathering even if any linear relationship were apparent. 

The final line of methodological criticism is that the rankings are so volatile over time that 
their reliability must be doubted. Considerable changes have been made to the methods 
used by rankers as their techniques have been evolving. A greater number of institutions have 
been included each year as rankings become more widespread. Corrections have been made 
progressively to data collection and reporting errors. The last three versions of theTHE -Thomson 
Reuters rankings illustrate the impact of these revisions, just for Go8 universities alone, blurring 
relative improvements in institutional capacity and performance (see Table 1 6). These data are 
subject to sampling errors, yet no variance measures or confidence intervals are reported. 


Table 16. Changes in the rankings of Go8 universities on theTHE rankings, 2010, 201 1 & 2012 


1 Go8 University 

2010-11 

201 1-12 

2012-13 1 

Melbourne 

36 

37 

28 

ANU 

43 

38 

37 

Sydney 

71 

58 

62 

Queensland 

81 

74 

65 

Monash 

178 

117 

99 

UNSW 

151 

173 

85 

UWA 

NR 

189 

190 

Adelaide 

73 

201-225 

176 


The problem of volatility arises in part from poor aggregation, without prior standardisation, of 
different performance indicators for the rankings (Longden, 201 1). Accordingly, a recent study 
comparing the rank order correlations of three different rankings - Webometrics,THEThomson- 
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Reuters and ARWU -found weak correlations below the top 40 institutions and volatility of ranked 
positions over time (Sawyer, 201 2). The authors concluded that the rankings are highly subject to 
the opinion preferences of the rankers: 

"For the rankings of universities, when we considered the three ranking measures for 201 1 , 
we found a very high rank correlation across the measures for the top ten universities. It 
would be difficult to imagine that any ranking measure could omit Harvard, MIT, Stanford, 
Oxford, Cambridge, Princeton, UC Berkeley from its top ten. 

But for universities ranked in the fifth decile, that is 41 st to 50th, it's an entirely different 
story. There's a very low rank correlation across the three measures in 201 1 ; that is, very little 
agreement as to the rankings, suggesting greater uncertainty and opinion dependence. 

But, more importantly, in the lower deciles, the volatility of rankings over time is appreciably 
greater, reinforcing the notion that these rankings are more uncertain. 

And for the lower deciles, cross-correlations over time across rankings are very low, again 
suggesting opinion dependence, rather than measurement. Universities are long-term 
institutions and their rankings should not change so guickly; nor should the divergence 
between measures change so guickly" (Sawyer, 201 2). 

An important implication of these methodological concerns is that shifts up or down in 
institutional positions do not necessarily reveal substantive changes in institutional performance. 
In this regard, two forces appear to be acting concurrently. One is a mostly upward force through 
gaming, whereby some institutions work to maximise their ranking scores by increasing inputs or 
outputs measured by the rankers. The institutions may do nothing different other than to report 
what they do differently (e.g. faculty/student ratios), or they may achieve a rise in an indicator 
value because they have paid particular attention to it (e.g. appointment of international faculty 
or highly-cited researchers). The other is a mostly downward force through slips in the rankings 
by one as a conseguence of gains by others rather than as a result of its own actions. The slippage 
may be a factor of, in the technical language of statistics, 'noise'- changes in contributing factors 
to scores that are not statistically significant or are redundant. A position in the rankings may be 
lost because of a fall in an institution's relative mean score on one or more of the sub-indicators 
used to derive the rank score. As noted above, the means of the indicator scores and the sets 
of scores whose means are aggregated (apart from Webometrics) are not standardised. Nor are 
standard error bars reported. 

The lay community in Australia is familiar with the error bar, or margin of error, of plus or minus 3 
points used in national opinion polling; if a party gains 2 points in its primary vote, we know not 
to assume that must involve a real change in voter sentiment, as the apparent gain is within the 
±3 points error range of the polling estimates. There is also some appreciation of the fact that the 
polls are backward-looking indicators rather than forward predictors. But with university rankings 
the noise in changes to institutional means is translated into actual shifts in rankings without 
caveats. Apparent year on year shifts of institutions in the rankings may be given disproportionate 
attention, especially as they carry a tail of legacy indicators embedded in their metrics. 

Improvements to rankings methods over the last several years notwithstanding, fundamental 
concerns remain: 

"No clear or universally-agreed measure of guality in higher education exists. Absent any 
consensus, each ranking combines an available set of observable indicators - each of which 
serves as a rough proxy for some factor notionally tied to guality - in a formula that can be 
guestioned both on the basis of its contributing elements and the manner in which the 
elements are combined" (Guarino et al, 2005). 
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The underlying concepts of rankings 

Noting the literature on rankingsTeichler (201 1) identifies (or more accurately opines) a range of 
stances towards them. He muses on 'sub-intellectual' views, whereby rankings are seen as "elements 
of a peep show; they stir up feelings of shame and desires for boasting and the like", with an 
acceptance of"primitive and highly ideological information about higher education" to the extent 
of a 'law'of negative correlation: "the lower the guality and the more biased the normative base, the 
higher the popularity of a ranking study is likely to be." More soberly, he argues that with the exception 
of attention to methodological flaws, world university rankings have been received uncritically, and 
are under-theorised, and that there is a need for deeper consideration of the underlying rationales 
for rankings, and the taken-for-granted assumptions of their producers and advocates. 

Such assumptions include (a) that there are common worldwide criteria for a good university, 

(b) that rankings induce healthy competition and stimulate improvement, and (c) that rankings 
contribute to greater transparency for guiding consumption and investment decisions. Alternative 
premises may be offered: (a) that different universities can be good at different things, and 
therefore evaluations of their guality should be referenced to the diversity of their missions; (b) 
that uni-dimensional vertical rankings can induce homogeneity, and therefore multi-dimensional 
ratings are necessary for reflecting and valuing horizontal differences; and (c) that different forms 
of information from those provided by whole-of-university rankings are reguired for the purpose 
of guiding student choice. 

Some attempts have been made to theorise rankings (e.g. Marginson & van der Wende, 2007; 
Ishikawa, 2009). Hazelkorn (201 1 ) identifies three explanatory foci: (i) the efforts of nation states 
(or supranational blocs such as the EU) to develop strategies for global competitive advantage; 
the adaptive efforts of higher education institutions to survive in a turbulent environment; and (iii) 
students and academic staff testing their self and peer perceptions of relative status. She suggests 
that these three foci align respectively, albeit in interacting ways, with globalisation and networks 
of power; organisational behavior and change; and social capital and positional goods. In part the 
analysis is derivative of the school of thought which regards education as social reproduction, as 
distinct from those schools which emphasise education as individual transformation and liberation, 
or as human capital formation. However, the analysis does not focus so much on students, their 
educational processes and their relationships, but rather on institutions and their relationships. It 
draws heavily on post-Marxist writers of the likes of Bourdieu, Castells, Foucault and Gramsci to such 
an extent that rankings are interpreted as agents of ruling class hegemony or as agents themselves 
preserving elite power relations. Curiously, in this world view, efforts to improve ranking methods, 
coverage and presentation are seen as acts of complicity in one's own subjugation. Such an analysis 
depends heavily on an inferential logic of inexorability and assumptions about relationships inherent 
to capitalist societies, even motives and conspiring drivers, which stretch the bounds of credulity and 
falsely imply passivity and weakness on the part of the ranked. 

As outlined above, world university rankings have many deficiencies. To the extent that the 
rankings have some value, at least for some users in respect of some institutions, it is sensible 
to work to make them as valid, robust and reliable as possible for those uses. It is also prudent 
to work on developing other sets of comparisons to complement the rankings, even for the 
research-oriented universities for which elements of the current rankings may have use value, and 
especially for other higher education institutions which are not at all well served by the current 
suite of indicators. An approach to theorising rankings which sees them as inherently inimical to 
some undeclared set of intrinsic values about universities does not provide any basis for advancing 
anyone's position. 
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The prevailing theoretical explanations fail to distinguish, on the one hand, between the explicit 
intent and the underlying function of ranking designs and, on the other hand, their uses and 
impacts. Efforts to sew these threads into a pattern may well be counter-productive. There is no 
single set of forces unfolding in some fashioned way, whether through the demonised forces of 
'neoliberalism'and 'new public management' nor even some less overt normalisation of behavior 
through 'anonymous surveillance of control'; rather, a number of dynamic often contradictory 
developments are underway. The contradictions have to do particularly with the operation of 
universities in markets and guasi-markets (Le Grand, 1991). Over-theorisation of more insidious 
underlying drivers of university rankings misses the main point: rankings are a means to serve 
commercial ends - both for the rankers and the ranked, but with the latter captured by the 
imperative of upholding and enhancing reputation, the currency of academic value. 

The guasi-market context of contemporary university functioning derives from the centrality of 
knowledge to modern economies and the changing role of the nation state. Knowledge is now 
more widely seen as a basis for innovation in enterprises and the economic development of 
nations. National economies are increasingly enmeshed globally, through government policies, the 
operations of firms, flows of capital and the movement of people. Governments have an interest in 
having sites of international standing for knowledge production, transmission and translation and 
for the formation of advanced human capital and technical know-how. They place an emphasis on 
applicability, accountability and measurable outcomes, and they prefer competition as a means 
of promoting value for money and spurring innovation. Higher education itself is also a growing 
business. Participation in higher education has moved beyond 'mass'and towards 'universal' 
participation. Student demand on a global scale is rising faster than the capacity of governments 
to provide at public expense and an increasing proportion of the cost burden is shifting to the 
private beneficiaries. 

University ratings and rankings, like Michelin guides to good restaurants and hotels, are a not 
unexpected feature of the growing business of higher education. Gross-national comparisons are 
an inevitable by-product of a more joined-up world. An increasingly competitive and globalising 
environment for universities - competition for talented students and staff and financial resources - 
necessarily puts a sharper focus on brand recognition. 

Importantly, the rankings are experimental, not in the sense of scientific inguiry, but in the 
commercial sense of exploring what market opportunities may emerge, and shaping and re- 
shaping them as need be. The rankings themselves are opportunistic: they respond to a rising 
demand for comparable information about universities; they use modern data collection and 
sorting methods to bring together some of the apparently comparable data that are available; and 
they seek to fill a void in the supply of easily accessible information about universities. 

The current main rankings are recent and not necessarily enduring. They are subject to 
competition from other suppliers and must maintain a threshold of credibility with their users. If 
they are opague and compromised, too narrow or biased, they will guickly lose support not only 
from students and other users but also from the universities subject to ranking. This is to recognise 
that rankings are being shaped not only by their producers but also by their uses and users. 
Gontrary to the inference of the hidden hegemony theorists, the participation of the ranked can 
be active rather than merely passive and represent strength of influence rather than submissive 
weakness. However, action to achieve better representation, balance and validity in comparisons 
of world universities needs to be informed by an understanding of the limitations of rankings. 

Rankings in general indicate an order of position. Rankings are standings ordered hierarchically. 

The position on a vertical ranking defines a relationship of relative merit (superiority or inferiority) 
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among those ranked. Typically they equate first with best and last with worst. However, any 
ranking is an arbitrary scale, and can exaggerate differences and apparent similarities, as there is 
no scientific precision for distinguishing institutions of close rank (Stella & Woodhouse, 2006). 

World university rankings in particular are selective (assessed against a limited set of criteria), 
norm-referenced (the score of one university is relative to the score of others rather than to pre-set 
quality criteria or standards) within a zero-sum game (gains and losses add to zero, a win for one 
means a loss for another, as those placed on top displace those below them). Rankings are also 
reductionist (simplistically presenting only parts of complex phenomena) and disproportionately 
value what is comparably measureable on the basis of available data. They have not only a 
descriptive or analytical role but a normative (preferencing) function in assigning universities in 
vertical relation to one another without accommodating horizontal functional diversity. 

Typically rankings do not indicate the intervals between first, second, third, fourth and so on. They 
do not necessarily reveal the extent of the gap between, say, a ranking of 5 and a ranking of 50 or 
500 in terms of performance standards. While some sense of measured differences can be gleaned 
from comparisons of mean scores on the sets of indicators comprising the aggregate ranked 
score, there is no test of statistical significance of the apparent differences let alone any necessary 
statistical validity to the aggregates themselves. Because the subjects of rankings are continuously 
modifying their performance, the reference points are constantly changing and judgements about 
quality can be inferred without clarification of the bases for judgement. Thus rankings have low 
heuristic utility. 

There are around 1 7,000 higher education institutions in the world (Rauhvargers, 201 1 ). The search 
for some discrimination among them on the basis of quality is not unreasonable. However it is 
not self-evident that a simple vertical ranking at the institutional level is an appropriate basis for 
qualitative discrimination, given (a) differences between institutions in their mission orientations 
and (b) differences within institutions in terms of capacity and performance, notably at the 
discipline level both for teaching and research. 

In this regard, the designer of the TH E-Thomson Reuters rankings has acknowledged that caveats 
necessarily apply to institutional rankings: 

"All global university ranking tables are inherently crude, as they reduce universities and all 
their diverse missions and strengths to a single, composite score. Anyone who adheres too 
rigidly to rankings tables risks missing the many pockets of excellence in narrower subject 
areas not captured by institution-wide rankings or in areas of university performance that 
are simply not captured well by any ranking" (Baty, 2012). 

However, such an admission is somewhat disingenuous, for the rankings are designed to show 
relative strength, and by so doing necessarily privilege scale over niche capacity. If the purpose 
of distilling information about universities' capacity and performance were to point out diverse 
pockets of excellence then ranking would not be selected as the method of comparison. 

A simple ranking on the basis of an aggregate sum of weighted available comparators also 
cannot properly reflect the diversity of judgements that are necessarily involved in any evaluative 
effort. When the focus is on the top 3%, or those institutions ranked in the top 500 (akin to the 
Fortune 500 rankings for enterprises), nuances are casualties. Even if such rankings might suggest 
something about the group of institutions that could claim to be positioned in an elite group 
(whether the leading 50 or 1 00 or 250) on a narrow range of criteria, they suggest nothing about 
the relative claims of the other 97% and the criteria by which they ought to be judged. Indeed, the 
'exclusion' of the majority, or their implied relegation, generates inappropriate pressures for them 
to emulate those that are ranked and to distort or compromise their own mission in so doing. 
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The major world university rankings are basically defining an elite club. They are doing so in a 
rather circular way given the selectivity of measures including inputs which naturally are skewed 
towards certain types of (better endowed) institutions. Putting this circularity aside, an analogy 
may be drawn with Olympic athletes, or 3-Hat restaurants, that perform in a class apart from the 
mainstream by virtue of surpassing threshold criteria which are set and re-set by the standout 
achievers. That is, there is a meritocratic dimension to university rankings alongside what may be 
construed as foundations of relative privilege. 

Yet in elite sports and fine dining, as in motor vehicles and fashion, there are marked functional 
delineations and separate bases for comparative assessments. Olympic athletes must pass an 
entry bar to competition but there are different bars for high jumpers and hurdlers, long distance 
runners and sprinters, swimmers and divers, gymnasts and shooters. Similarly, unless one has 
a peculiar palate, there is little sense in comparing Chinese, Indian, Italian, Japanese, Ottoman, 
Peruvian and Thai food on a single ranking to find the world's top restaurants. Nor is it sensible to 
compare 4-cylinder, 6-cylinder and 8-cylinder motor sedans and wagons on a single composite 
rank that accounts for handling, safety, design, comfort, accelerative speed, torque, energy 
efficiency and price. 

Assessments of relative quality, performance and value for money need to be related to purpose 
and role. Variety in needs and tastes alongside diversity in products and services necessitate a 
plurality of means of comparative assessment. That is why ratings tend to be used much more 
than rankings in consumer guides or, at least, different rankings are used for different comparisons. 
To the extent that just one dominating set of rankings, which are predicated upon a strong 
research role for universities, imply a policy preference for a stratified view of higher education 
they may well function to diminish the value of horizontal diversity (Proulx, 2007, Teichler, 2011). 

This is not to suggest that a ranking such as the ARWU is totally invalid or inappropriate. To the 
contrary, it is a reasonably useful benchmarking tool (excepting the lottery of laureates) for the 
set of elite research universities that seek to have their strengths acknowledged and augmented. 
However, like the Olympics, it is not a competition for the majority of universities worldwide, 
not least for high-performing universities whose strengths are to be found in their teaching and 
service missions rather than research. 

Similarly, the Leiden rankings are useful for research-oriented universities, although they capture only 
knowledge discovery not its translation. The THE-Thomson Reuters and QS rankings, however, seem 
to have confused purposes, being neither strong enough in assessing relative research strengths nor 
sufficiently direct and rigorous in assessing relative strengths in education. Nor do any of them give 
serious attention to the'service'or'engagement'functions of contemporary universities. 
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The uses and impacts of rankings 

Despite their deficiencies, world university rankings are making an impact on institutional 
behaviours and government policies. Why do they do so? Three basic reasons may be suggested. 

First, university rankings fill a void. Simple guides are wanted by people having to make choices 
faced with an ever increasing complexity of university functions, organisational arrangements 
and a proliferation of data including marketing hype. It may be argued further that rankings 
facilitate the identification of important entities, highlight their principal attributes, and enable 
efficient search and sort operations. Indeed, rankings may be one of the few means of navigating 
ambiguous data sets that include a mix of cardinal and ordinal scores, have many missing 
information elements, lack balance in guantity and guality across entities and criteria, and rely 
on complex networks and sources (Xiaoye et al, 201 1). Importantly, universities rely internally on 
tacit rather than explicit judgements of quality and have failed to communicate clearly, to the lay 
communities that sustain them, what they do and what they stand for. However, what is made 
explicit can be limiting: 

"Because it is so difficult - for insiders and outsiders - to judge the quality of a university's 
technology and outputs, it is often more cost-effective and useful for institutions to copy 
the structures and behaviors of universities perceived as being successful" (Morphew & 
Swanson, 201 1). 

Second, rankings appear to be based on 'objective' assessments which are independent from 
the universities being ranked. They appeal to the media and, arguably, to 'consumers' of higher 
education services, including students (international if not domestic) and parents, as well as 'end- 
users' such as employers of graduates. However, they are content-poor sources of information for 
students, and obscure more than they reveal about the breadth and depth of university offerings. 
Additionally, as the indicator-based assessments are'over-simplified'or'under-complex', action 
taken by universities in response to them may fall short of the actual challenges of increasing 
complexity which confront contemporary universities (Altbach, 2006). 

Third, university rankings resonate in an academic culture of fierce individual competition and 
status seeking. Being the first to discover, to comprehend, to pioneer or even to fault - and to be 
credited for so doing, eventually if not immediately - that is a key driver and prize of academic 
achievement. Rising in the ranks becomes a motive in itself because it conveys progress and 
confers esteem. The rankings can become drivers by defining what quality means and shaping 
university mission and balance of activity (Marginson, 2007a), inflating the 'academic arms race' 
(Tierney & Hagedorn, 2002), inducing universities to chase ever more resources (Ehrenberg, 2004), 
intensifying competitive pressures, establishing as a worldwide norm the science-strong research 
university of the Anglo-American tradition, and giving emphasis to institutional stratification and 
research concentration (Marginson, 2007b). 

The competition among rankers may also be seen as one of the factors shaping institutional and 
government responses: 

"The competition between ranking agencies is then like a tournament of portfolios, and 
their objective is to persuade observers to converge to their portfolio. They want observers 
to be anchored by their ranking. When a proposed measure becomes widely accepted, 
universities minimise risk by adopting practices maximising that ranking" (Sawyer, 2012). 

However, institutions and governments have their own purposes and for which ends they seek 
to employ the rankings as means. Continuous improvement stands out as a major purpose, 
and the exposure that world university rankings give to local institutional performance against 
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international benchmarks adds impetus to the improvement agenda, both on the part of 
university and government executives. In this endeavor there is also the risk that the means of 
measurement become confused with the ends of guality higher education (Dill and Soo, 2005) and 
that an institution will try to succeed according to the indicators which themselves can distract it 
from attending to its real challenges, special opportunities and underlying problems. Additionally, 
the rankings game is a risky and unpredictable one: 

"Rankings are a game everyone plays, but a game with constantly shifting rules that no one 
can control (Morphew & Swanson, 201 1). 

Nevertheless, rankings may be seen as having various uses or to serve various purposes, whether 
designed explicitly to be used for such purposes or not. The major uses are: 

i. influencing consumer choice, including information and guidance for prospective students, 
and institutional marketing 

ii. benchmarking for improvement, including setting targets and introducing process changes 
based on practices of highly-ranked institutions 

iii. partner selection, including recognition of gualifications or prior study for credit, and choice 
of institutions for collaboration in research and research training 

iv. informing investment decisions, including institutional actions to attract industry and 
benefactor funding, and government decisions to allocate funds for special programs 
and centres or to skew allocations through performance incentives linked to rankings. 


Institutional uses and impacts of rankings 

Universities tend to have love-hate relationships with rankings; not only do they hate them when 
they don't look good on them but love them when they do, they also are ambivalent towards 
them, sceptical of their validity but intimidated by their potential to do harm: 'we know they are 
dubious but we have to live with them';'we may not like them but they're not going away';'we 
may dismiss them but others are using them and we can't afford to be missed out'. 

Universities may variously make use of rankings for marketing purposes - marketing to 
prospective students, marketing for staff recruitment, marketing to prospective investors, 
benefactors and sponsors, and general brand positioning. They may also use them to evaluate 
guality of performance, and identify priorities for resource allocation, either building on strengths 
and potential or closing down poor-performing areas. 

Some universities have dedicated staff for managing their rankings, correcting omissions and errors 
in ranker data, watching changes in ranker methods, modelling the range of effects of performance 
shifts on individual indicators, harnessing institutional actions to improve performance on the 
indicators that might have a positive impact on institutional positioning, and canvassing other 
institutions and possible peer reviewers with a view to raising their reputational scores. 

Much attention has been given to negative behaviour associated with rankings, such as cheating 
and gaming. A recent article in the NewYorkTimes exposed misleading reporting and other 
manipulations in the US national rankings context: 

"Iona College in New Rochelle, north of New York City, acknowledged that its employees had 
for years provided false information not only about test scores, but also about graduation 
rates, freshman retention, student-faculty ratio, acceptance rates and alumni giving. Other 
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institutions have found ways to manipulate the data without outright dishonesty. In 2008, 
Baylor University offered financial rewards to admitted students to retake the SAT in hopes 
of increasing its average score. Admissions directors say that some colleges delay admission 
of low-scoring students until January, excluding them from averages for the class admitted 
in September, while other colleges seek more applications to report a lower percentage of 
students accepted" (Perez-Pena & Slotnik, 201 2). 

There appear to be fewer opportunities for cheating on the world rankings. Rather than cheating, 
there is some risk of manipulation of student/staff ratios, recording of affiliate publications, only 
reporting data relating to specific campuses, and canvassing of peer references. Such actions are 
more likely to affect results in QS and THE- Thomson Reuters than in those rankings based on 
more objective output measures. With regard to the current suite of international rankings, the 
main problem is the pressure they put on lower ranked and unranked institutions to emulate 
inappropriately those ranked highly. 

Surveys of institutional responses to rankings suggest that many institutions are spurred to 
actions such as incorporating the outcomes of rankings and setting targets in strategic planning, 
benchmarking to identify weaknesses and eradicate problems, and structural reorganisation to 
achieve higher rankings (Hazelkorn, 2007). For many of the less research-intensive institutions, 
setting targets to achieve a ranked score may be beyond the bounds of practicality and efforts 
to that end may well involve counter-productive initiation or curtailment of activities. For others 
there may be useful performance gains to be made from rankings giving rise to a reconsideration 
of strategic focus, operating practices and measures of success. 

A study by the Higher Education Funding Council for England (HEFCE) found that rankings could 
also serve as leverage to make higher education more effective and innovative (HEFCE, 2008). 
Rankings were seen to be helpful in the following areas: 

• improved evidence-based decision making - rankings can prompt institutional discussions about 
what constitutes success and how the institution could better document and report that success; 

• wider conversations about monitoring and reporting success - encouraging institutions to 
move beyond their internal conversations to participate in broader national and international 
discussions about new ways of capturing and reporting academic success and excellence; 

• identification and replication of model programs; and 

• increased institutional collaboration. 

The second last finding - copying practices of highly ranked institutions - is double edged. On the 
one hand it can provide a benchmarked basis for institutional improvement. On the other hand 
it can lead an institution away from its own mission and distinctive values, and act generally to 
reduce diversity within the higher education system. The copying approach can lead to greater 
attention being given to research than teaching even where an institution has few areas of serious 
research. Action may include poaching of star researchers, and directed internal investment to 
build research capacity in particular areas. 

The last finding of the HEFCE study - increased institutional collaboration - is interesting in view 
of the criticism of rankings as an instigator of competition among institutions to the detriment of 
collaboration. A study by the Institute for Higher Education Policy also suggested that rankings foster 
collaboration, such as research partnerships, student and staff exchange programs, and institutional 
or faculty alliances (IHEP, 2009). At the very least, rankings can function as starting points to identify 
institutions with which to collaborate and partner, and weigh up collaborative options. 
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The implications of rankings for public policy 

In general, world university rankings give greater public visibility to higher education issues and 
institutional performance. They have also given rise to three relatively new phenomena in public 
policy: governments targeting the availability of scholarships for their nationals to highly ranked 
universities internationally; governments limiting the partnering of their home universities to only 
those foreign universities that are highly ranked, and governments aspiring to have a number of 
their universities highly ranked or otherwise designated 'world class'. 

Rankings for scholarship funding 

Mongolia, Kazakhstan and Qatar offer scholarships only to those students who gain admission to 
the top 100 ranked institutions. The Netherlands gives priority to immigrants with gualifications 
from the top 1 50 universities. Russia's study-abroad scholarships funded via its $150 million Global 
Education program are to be targeted to top-ranked universities. The scholarships provided by the 
Government of Brazil {Science Without Borders) and Ghile {Becas Chile) are targeted to the top 1 50 - 
500 universities on world rankings. Ecuador has two sets of scholarships: one for students accepted 
by institutions in the top 50 and another for institutions in the top 1 50. Peru and Golombia, 
similarly, are developing scholarship programs designed to encourage their students to study 
at highly ranked international universities. In the case of Peru, the Government has responded 
to increasing industry demand from graduates by announcing that 1 ,000 new postgraduate 
fellowships would be made available by 2016, as well as 1,500 scholarships for Peruvian students 
at foreign universities. The study abroad incentives also reflect a concern that none of Peru's 90 
universities are ranked in the top 500 of any of the major rankings. 

These decisions of 'sending' countries to target or restrict scholarships to universities highly ranked 
on international comparisons generate incentives for universities in 'receiving' countries to achieve 
good rankings. Eor Australian universities, which have a high level of dependency on income from 
international students and which are generally seeking to diversify the source countries of their 
international enrolments, there can be much at stake in securing growth in student flows from 
Latin America, Asia, the Middle East and Africa. 

Limiting authorised partnering to highly ranked foreign universities 

The Government of Singapore has applied world university rankings to identify foreign universities 
whose gualifications they will recognise in Singapore, to screen foreign universities seeking 
to establish campuses in Singapore and to limit authorised partnerships between their home 
universities and foreign universities. The Government of India has restricted joint degree programs 
with Indian institutions to universities ranked in the top 500 on the ARWU or TEI E-Thomson Reuters 
rankings. Eor Australian universities seeking a commercial presence in Asian markets it becomes 
necessary to achieve a threshold position on the world university rankings. 

Aspiring to have world class universities 

Some governments are inclined to respond to educational rankings by setting 'targets' to raise 
national performance - such as declaring a goal of being ranked within the top group of countries 
on the OEGD's Programme for International Student Assessment (PISA) scales - irrespective of 
their merits, educational impacts and opportunity costs. In like manner, several countries have 
stated goals of establishing or increasing the number of universities placed on world university 
rankings. As Jamil Salmi has pointed out, for many developing economies such a strategy may be 
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misguided: "the hype surrounding world-class institutions far exceeds the need and capacity for 
many systems to benefit from such advanced education and research opportunities, at least in the 
short term" (Salmi, 2009). 

At the same time, there is a noticeable trend on the part of long-developed and newly-emerging 
economies to shore up their capacity for advancement in knowledge and know-how at the 
top of their national higher education systems. Germany launched its Exzellenzinitiative in 2005 
with the aim of concentrating investment in graduate schools, research clusters and forward- 
looking centres of excellence. In 2008 France announced the PRES initiative {pdles de recherche et 
d'enseignement superieur) to develop ten internationally competitive university clusters. Denmark, 
India, Indonesia, Japan, Malaysia and South Korea have also taken steps to restructure their 
systems and institutions to align with indicators used in the rankings. In 2004, Taiwan set out 
detailed goals and investment priorities: 

"The specific objectives are to be set as ten elite (or Asian top-tier) universities or fields of 
research within five years, and at least one university to ascend to the world class (within, 
for example, the top 1 00 universities in the world, or, by comparison, the top 80 in the US) 
within ten years" (Development Plan for World Class Universities and Research Centers of 
Excellence, Taiwan, 2004). 

Earlier the People's Republic of China had determined to concentrate investment in a small 
number of its major universities, through the 21 1 program announced in 1 995, initially designed 
for 100 but now involving 106 universities and colleges, and the 985 program announced in May 
1 998 and involving just nine universities and institutes, with the aim of raising their research 
capacity, performance and international status. President Jiang Zemin declared: "China must have a 
number of first-rate universities of international advanced level". In 201 2, the Chinese Government 
increased spending on science and technology by 1 2.4% to 229 billion renminbi ($36.2 billion). 

These developments in the concentration of investment in leading research universities 
and clusters work to intensify the international competition for talent, ideas and innovative 
applications. As university capacities increase in Asia, Europe and Scandinavia and elsewhere, 
Australia too must intensify its efforts in order to keep up with the pace-setters. Countries needing 
a step-change in their research competitiveness cannot afford a step-by-step dilution of their 
research investment. 


The importance of highly-ranked 
research universities 

One characterisation of the class of world leading universities - the'emerging global model'(EGM) 
- identifies the following eight factors: an institutional mission that transcends the boundaries 
of its home nation; increasing intensity of knowledge production; student and academic staff 
recruitment on a worldwide basis; academic roles geared to high performance and productivity; 
a diversified funding base; government facilitation of university-industry collaborations; complex 
organisational structures including semi-autonomous centres and institutes; and global 
collaboration through alliances and networks (Mohrman et al, 2008). Nevertheless, the strategic, 
cultural, and organisational factors identified for the EGM cannot be effective without sufficient 
scale of inputs. 
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Universities of the EGM are high volume producers of high guality knowledge outputs, including 
highly cited journal articles and other channels of communication. One might ask: 'why are the 
articles highly cited'? Normally the answer is because they are sources of ideas breakthroughs 
or discoveries, on the basis of which knowledge is gained. It just happens, by and large, that the 
best outcomes are produced by the best placed institutions, and that a threshold level of input 
resources is reguired typically for university to be among the best placed. One such threshold is 
an asset base exceeding $1 billion, alongside a scale of high guality talent in the student and staff 
body. Not all rich universities are outstanding but given the costs of doing breakthrough research 
poor ones cannot be. While there are excellent outcomes produced in some institutions which do 
not meet all the threshold criteria, they are not of an eguivalent scale or degree of consistency. 

To sustain economic competitiveness, countries like Australia must be able to generate new 
knowledge and understand and interpret new knowledge generated elsewhere; they cannot 
rely on a strategy of passive absorption of foreign technology. To benefit from the public good of 
world knowledge, nations have to be actively engaged in cutting edge research. Free riding on 
the rest of the world's research is not a realistic option - because the links between researchers 
are personal and they are based on informal trading in ideas, technigues and devices. To access 
and make sense of basic research you have to be a contributing insider to the community of 
international researchers in a field. The capacity to understand and use the results of basic research 
performed elsewhere reguires a considerable investment in institutions, skills, eguipment and 
networks (Pavitt, 2001). 

The need for research concentration within national higher education systems can be understood 
to be driven by concerns to (a) strengthen and integrate capabilities in order to address complex 
and pressing national problems, including those with global dimensions; (b) increase the 
international visibility of national strengths for attracting talent and inwards investment; (c) open 
up opportunities for collaboration with universities in other countries that have similar reputation 
for excellence; (d) and ensure sufficient capacity for cutting-edge research for the nation through 
access to world knowledge developments. These motives reflect a view of contemporary 
universities as engines of innovation and economic development and sources of solutions to 
social and environmental problems. 

One of the drivers of innovation is the clustering of talent and the production of new knowledge. 
Large research-intensive universities are among the most effective aggregators of highly gualified 
personnel (Usher, 2009). In many fields of research in the natural sciences there is a 'critical mass' or 
threshold effect, and "large, well-funded and well-led research groups produce more publications, 
of higher impact, and receive much higher international recognition than do smaller groups" 
(NBEET, 1993). There are several advantages of scale in research, as noted in a EIEFCE (2000) review 
of research funding policy: 

"A larger group of researchers adds to overall vitality, through peer stimulus, the opportunity 
to exchange and develop ideas, and to be spurred by visible achievement. Second, the per 
capita marginal costs of research (administration, clerical support, etc) are reduced when 
a larger group contributes to infrastructure. This factor is significantly accentuated by the 
high cost of major eguipment and facilities in the experimental sciences. Third, larger groups 
make possible the simultaneous and parallel development of research themes, leading to 
an overall acceleration. Fourth, group size contributes to diversity of thought and of sub- 
discipline, increasing the likelihood of cross-fertilisation and fruitful development. Fifth, 
larger groups of research students provide a more supportive atmosphere for research 
training" (EIEFCE, 2000). 
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It is generally accepted that new knowledge can move enterprises into leading market positions 
beyond the point where existing technology can be imported and adapted (Litan & Mitchell, 

2008). In the guest for innovation it is the ability to marshal resources, including intellectual 
capability, to achieve 'significant advances'ahead of the competition that counts. 

Additionally, the capacity of universities to undertake long-term research underpins their direct 
and indirect contributions to Australia's national innovation system. The research literature 
indicates that basic research is an important source of (i) the skills (particularly those based on tacit 
knowledge) required to translate knowledge into practice (Salter & Martin, 2001) (ii) an enhanced 
ability to solve complex technological problems, and (iii) the 'entry ticket' to the world's stock of 
knowledge, providing the ability to participate effectively in networks and absorb and exploit the 
resulting knowledge and skills (Martin &Tang, 2007). Importantly, basic research, or long-term 
research whether "curiosity-driven" or "use-inspired" (Stokes, 1997) which explores underlying 
issues, underpins disciplinary advancement. 


The importance of excellent universities 
serving national and local needs 

While it is necessary to strengthen the research performance peaks of the system it is not sufficient 
to do only that, particularly given the concurrent and larger imperative to widen higher education 
participation and raise the educational attainment and skill levels of the workforce. The double- 
edged challenge for public policy is to strengthen the leading institutions without impoverishing 
the core of the system which caters for mass participation, and to enlarge the system equitably 
without diminishing the quality of the elite performers. 

There is the additional challenge of encouraging universities to contribute in varying ways to 
national and local objectives: economic, social, environmental and cultural. 

With regard to economic objectives, discovery drivers of innovation can be paradigm-shifting but 
science-push innovations in business are exceptional, typically lagged and rarely linear. The great 
bulk of innovation, involving enterprises competitively responding to demand drivers, derives 
from clever modifications of existing knowledge broadly embedded in the creative, technical and 
organisational skills of populations (Cutler, 2008). Hence it is necessary to recognise the importance 
of research of national, regional and local relevance that is highly valued by end users. It is also 
essential to focus on the major contribution that universities make to national innovation: their 
graduates. This is not to suggest that the metrics used in rankings for research universities should 
be discarded or replaced but rather to call for them to be complemented by other sets of indicators 
reflecting the varying roles of different universities and other higher education institutions. 

With regard to environmental objectives, for instance, the dimensions of the problems are 
frequently global as well as local. It Is not unreasonable to expect that research oriented to solving 
local environmental concerns should not only satisfy local customers but also pass muster in the 
internationally peer-reviewed literature; otherwise there is no basis for local confidence in the 
quality of the research. 

Universities of the EGM class undertake research comprehensively in two senses: research across 
the disciplines and cross-disciplinary subjects they teach - both liberal and professional courses; 
and research across modes and orientations of inquiry - both basic and applied research. Other 
universities may be narrower in the range of fields in which they conduct research, and they may 
orient their research in particular ways and to particular ends. 
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The world university rankings predominantly use research metrics linked to the outputs of basic 
research. In this context, some express concern that the pressure to align institutional behaviours 
with the rankings measures will lead to a reduction of more socially relevant and commercially 
useful research (Hazelkorn, 2009). Such concerns imply a need to develop indicators to capture the 
contributions to innovation through human capital development and knowledge generation and 
application that are not captured by the metrics used in the current set of world university rankings. 

In the context of governments seeking to establish 'world-class universities' or strengthen their 
leading national institutions through concentrated investments or structured collaborative 
arrangements, such as clusters and mergers, guestions arise about the structural balance of 
national higher education systems. Yet we are still awaiting answers to the guestion posed by 
Jamil Salmi: 

"Are only research universities world-class, or can other types of tertiary education 
institutions (such as teaching only universities, polytechnics, community colleges, and 
open universities) also aspire to be among the best of their kind in an international 
perspective?" (Salmi, 2009). 

In Australia, the current Government policy on the structural design of higher education 
limits even such a discussion. Polytechnics were prematurely disappeared during the Dawkins' 
unification of the late 1 980s/early 1 990s. Curiously and without explanation, the Bradley panel 
in 2008 explicitly rejected them as a structural option for the future, whereas other countries 
(e.g. Finland, Singapore) have built their national innovation systems around deliberately 
differentiated higher education systems, including specific and separate roles for polytechnics 
and universities. Community colleges, to the extent their function is available at all, are a residual 
part of a withering vine of Technical and Further Education colleges and blurring with the bottom 
end of admissions to larger suburban and regional universities. Australia has favoured a mixed 
mode approach to the use of on-line education in contrast, for instance, to South Korea which 
has 1 8 cyber-universities. Fligher education continues to be supplied predominantly (some 90% 
of student enrolments) by public sector institutions, with the private sector in Australian higher 
education one of the lowest in the world. Until there is greater diversity on the supply side of the 
Australian higher education system, reliance on world university rankings that place a high value 
on research performance, as the principal reference point, will tend to diminish diversity. 

Teaching only universities - whether on the elite model of the US liberal arts colleges or larger mass 
higher education institutions - have been defined away by the narrow 'provider category' types and 
standards under the TEQSA regimen, where all universities must do some 'research'. The advantage 
of theTEQSA approach is that universities are distinguished from other provider types, and that 
distinction underpins difference of policy treatment in regulation and funding between universities 
and other 'higher education providers'. Flowever, there are as yet no 'threshold research standards' 
setting out expectations of the university research function, and it will be difficult to pre-set such 
'standards' without constraining the ways and means by which research may be performed and the 
purposes for which it is done. Unless work is undertaken to outline the array of research orientations 
there is a particular risk that TEQSA's standards for research will be based on the traditional models 
of basic research conducted predominantly in the research-intensive universities. 

In the event that a few established universities are unable to satisfy the TEQSA criteria for research, 
the only option currently available would be for them to forfeit university title and become merely 
a'higher education provider'. That seems a harsh outcome which could arise simply for want of 
diversity in the allowable nomenclature for higher education institutions in Australia. The prefix 
'teaching-only' is problematic in connoting that an institution 'only does teaching'. Perhaps other titles 
could be explored, e.g. community university, institute, polytechnic or university of applied sciences. 
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As they are currently structured, world university rankings do not relate well to the missions of 
universities whose principal mission is not research, or at least not internationally-referenced basic 
research. There are important roles for universities that focus on producing guality graduates for 
Australian labour markets and providing services, including research translation, testing, survey 
and other services for businesses and community groups. In this regard, institutional positioning 
strategies pulled by the current suite of world university rankings which privilege academic 
research can limit institutional vision and distort mission, and reinforce sameness in aspiration and 
benchmarks of success. 

An important challenge is to develop diverse reference points. For that to happen, three 
developments will be necessary. First, there will need to be sharper clarity of mission orientation 
on the part of different universities and other higher education institutions. Second, work will 
need to be done on developing a richer array of education and service oriented indicators of 
university capacity and performance. Third, attention will need to be given to describing the 
array of research orientations. 

Martin Trow identified some time ago that it is unreasonable, unfair and inefficient to place 
expectations on institutions to become what they are not set up to be: 

"A central problem for higher education policy in every modern society is how to sustain 
the diversity of institutions, including many of which are primarily teaching institutions 
without a significant research capacity, against the pressure for institutional drift toward 
a common model of the research university - the effort alone shapes the character of 
an institution to be something other than what it is - a prescription for frustration and 
discontent" (Trow, 2003). 

The problem with the Trow approach is that it sets up a negative definition for the majority of 
higher education institutions: they are defined as not having significant research capacity, rather 
than as having focused and locally relevant research capacities and highly-developed teaching 
capacities and focused community service capacities. This is not to suggest that research-intensive 
universities do not or cannot have strengths in teaching, research applications and community 
engagement. Rather it is to point to the failure of policy makers and institutions to come to 
grips with the scale and diversity of contemporary higher education and create a language for 
expressing its richness. 


Why are some Australian universities 
doing as well as they are, and what will 
sustain their performance improvement? 

The TFI E-Thomson Reuters rankings editor has painted a grim future for English universities (such 
as Birmingham, Bristol, Leeds, Newcastle and Sheffield) whose positions deteriorated in the 
201 2-1 3 TFIE rankings: 

"Outside the golden triangle of London, Oxford and Cambridge, England's world-class 
universities face a collapse into global mediocrity. Fluge investment in the top research 
universities across Asia is starting to pay off And while the sun rises in the East, England 
faces a perfect storm: falling public investment in teaching and research; hostile visa 
conditions discouraging the world's top academics and students from coming here; and 
a serious uncertainty about where our next generation of scholars will come from, with a 
policy vacuum surrounding postgraduate study" (Baty, 201 2b). 
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One might point to the political nature of such a commentary, and its unabashed nationalist 
sentiment, to suggest bias in the construction of the rankings that give rise to the cause for such 
comments. Nevertheless, given the changing economic circumstances of nations, we can expect 
a gradual flattening in the representation of US, UK and European institutions in world rankings 
as public investment in higher education and university research there is diluted. Particular 
challenges confront US public research universities across several states which are not only cutting 
back on their financial support for research universities but also seeking to curtail scholarship and 
research in several fields. The philistines are at the door: Governors of several US States appear 
to be confronting the academy in a more joined-up way than previously, guestioning the use- 
value of investment in fields of education and research which do not appear to have immediate 
application or which are emblematic of liberal values. 

Concurrently, there is an agenda for greater intensity of research performance in France, Germany, 
Scandinavia, Switzerland and the Netherlands as investment in research is more concentrated in 
their top universities. Future growth in capacity and performance can be expected also from the 
leading universities in the stronger economies of Asia (notably China, Singapore, South Korea), the 
emergent Asian economies (notably India, Indonesia, Malaysia, Thailand and Vietnam) and Fatin 
America (notably Brazil, Chile and Mexico). 

Australia's relative improvement in several of the world university rankings reflects the plateauing 
of inputs to US and UK universities, the lagged accounting for the emergent Asian universities, 
alongside some recent absolute lifts in funding inputs for Australian universities. With reference 
to the 2012 ARWU rankings, with 19 of Australia's 37 public universities making the top 500, the 
Minister for Tertiary Education asserted: 

"This outstanding result is due in large part to the record $37 billion the Government has 

invested in our universities, as well as the hard work of Australia's universities," (Evans, 201 2). 

It is true that increased funding for universities has resulted from the present Government's 
decisions post-2008 to improve the indexation of payments to universities for recurrent purposes, 
including research activities. Additionally, the Rudd Government made a very high capital 
injection into universities, including for research infrastructure, as part of it economic stimulus 
measures in 2009-2010. 

Nevertheless, the apparent rise of some Australian university positions on the rankings cannot be 
attributed simply to increased government funding in recent years. Indeed universities generally, 
and Go8 universities in particular which lead the Australian rank order on all measures, have not 
enjoyed real terms growth in per-student revenue from the Australian Government since 2007, the 
improved indexation of recurrent grants notwithstanding. A major loss to income, especially for 
Go8 universities, resulted from the Fabor Government's decision to prohibit the enrolment of full- 
fee domestic undergraduate students after 2008. 

Over recent years, Australian universities have experienced income growth from international 
student fees, philanthropy and industry funding of research. The funding from these sources is also 
factored into the rankings indicators. 

Welcome increases in funding for the indirect costs of research, rising from 20 cents to 30 cents 
in the dollar, and the large capital injection in research facilities have influenced to a small degree 
only those rankings that give weight to input metrics, such as the TFI E-Thomson Reuters rankings 
which include industry income, research income and income per academic staff In future years 
the funding increments may affect rankings indicators despite diminished discretionary income of 
Go8 universities in particular as a result of the Government reguiring matched funding for capital 
projects and failing to provide for recurrent maintenance costs. 
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Australia's universities will still have to find at least 70 cents in the dollar for the indirect costs of 
the research grants they win through national competitive processes. Those grants typically pay 
only around 70% of the direct project costs and, unlike research funding schemes in the US and 
elsewhere, do not provide for principal investigator salaries. 

In its October 201 2 mid-year budget review, the government reneged on it prior commitment to 
raise funding for indirect costs to 40 cents in the dollar. Some $150 million annually will be lost, 
mostly by Go8 universities. 

Additionally, as reports commissioned by the Government itself have found, Australia's universities 
continue to be impacted adversely by shortfalls in funding for domestic undergraduate students 
(Bradley, 2008; Lomax-Smith, 201 0). Gonseguently, the universities have a very high level of 
dependency on fee income from international students - both to make up shortfalls in funding for 
domestic students in higher-cost disciplines and to compensate for under-funding of research. 

The Howard Government (1 996-2007) was a strong supporter of research. The average increase 
in Gommonwealth competitive research grants over the period 1 997-2007 in real terms was 7.1 %, 
primarily due to significant increases in ARG and NHMRG recurrent funding via Backing Australia's 
Ability initiatives and the response to the Wills report. Figure 7 shows the increase in real terms of 
university income from Australian Gompetitive Grants over the period 1 992-201 0. 


Figure 7. University income from Australian Competitive Grants, 1992-2010, in 2010 Dollars 



Source: Higher Education Research Data Collection, various years 


Figure 8 shows the annual percentage change in real (Gommonwealth) income, 1993 to 2010. The 
average annual change under the Keating Government (1993-96) was 5.3%. Under the Howard 
Government (1 997-2007) the average annual increase was 7.1 % (with a spike over 2003-2007 of 
1 0.1 % per annum) compared with merely 3.2% over 2008-201 0 under the Rudd Government. 
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Figure 8. Annual increases in university income from Australian Competitive Grants 
(constant dollars) 



The Labor (Rudd and Gillard) governments have concentrated on funding short-term initiatives 
focused on infrastructure and capital funding as part of the stimulus package and terminating 
programs such as Future Fellowships, Super Science Fellowships and National Collaborative 
Research Infrastructure Scheme (NCRIS) projects. 

Under the Rudd Labor Government the rate of annual increase in Australian Competitive grants fell 
to 3.2% over the period 2008-10, and while this includes a short-term increase due to the funding 
of Super Science and Future Fellowships, the annual increase will begin to decline rapidly from 
2014 as a result of the termination of the Future Fellowships Scheme. The Labor Government has 
also not provided for the further funding of major national research facilities. 

Over the period 2002-1 2, the average funding for the first year of ARC Discovery grants declined in 
real terms by 8.9%, peaking in 2006. Over 2002-2006 there was an 8.5% increase while over 2007- 
201 2 there was a decline of about 1 3%. In the same period success rates for Discovery fell from 
25.47% to 21.95% after a peak in 2005 under the Coalition Government of 30.9%. 

Funding for the Research Training Scheme (RTS) has not been increased (other than by annual 
indexation) since its introduction in 2001 . Conseguently, RTS funding in real terms has declined by 
about 14% in the period 2001 and 201 2. At the same time the Domestic FIDR load has increased 
significantly so that RTS funding per Domestic FIDR EFTSU has fallen by over 23% in the same period. 

Similarly block funding through the Institutional Grants Scheme/Joint Research Engagement 
scheme (IGS/JRE) has declined in real terms over the same period. In addition the Labor 
Government has changed the funding formula to remove Category 1 (national competitive 
grants) funding from the JRE block fund and in doing so has reduced the Go8 funding share. The 
change to the JRE formula was partly designed to offset the gains expected to be made by Go8 
universities through the application of research guality evaluation results from the Excellence in 
Research for Australia (ERA) initiative. 

The ERA evaluative process itself from 2009 and subsequently has provided incentives, 
reputational as well as financial, for universities to address areas of weak research performance and 
build on their research strengths. Action has included buying in star researchers and expanding 
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doctoral student enrolments, especially international students. This sharper focus on research 
performance may well have influenced factors taken into account by the world university rankings. 

The 201 0 Excellence in Research for Australia (ERA) outcomes provided the Government with the 
opportunity for the first time to provide research block grant funding based on demonstrated 
research excellence. In 201 2 the Government incorporated those outcomes into the funding 
formula for the Excellence Gomponent of Sustainable Research Excellence (SRE) Threshold 
2 funding. The funding driven by ERA outcomes in 2012 was $65.7 million of SREThreshold 
2 funding, only about 4% of the total funding provided through research block grants. The 
weightings used in the formula were 7:3:1 :0:0 for ERA rankings of 5:4:3:2:1 . Australia's investment 
through ERA needs to be compared with the English approach through its Research Assessment 
Exercise (RAE). There the amount allocated for research excellence annually exceeds £1 billion, and 
the top rated research is weighted by a factor of 9 rather than 7 as in Australia. Nevertheless, the 
ERA weights sent a strong signal to Australian universities to strive for excellence in research. 

As noted above, Go8 universities lead the Australian placements on all the major world rankings. 
This is largely because the rankings are skewed in favour of academic peer-reviewed research output 
guality and the Go8 universities perform the lion's share of fundamental research, and especially in 
health and medical research including clinical research. Table 1 7 shows that the Go8 accounts for 
almost two-thirds of all of Australia's higher education research effort, including more than half of 
applied research and experimental development, and some three guarters of basic research. 


Table 1 7. Australia's Higher Education research effort by activity type, 201 0 


1 Type of activity 

National Higher Education Effort ($m) 

Go8 share (%) I 

Pure basic research 

2,052.2 

(25%) 

82 

Strategic basic research 

1,653.0 

(20%) 

60 

Applied research 

3, 835.8 

(47%) 

52 

Experimental development 

662.0 

(8%) 

60 

Total research activity 

8,203.0 

(100%) 

62 


Source: Australian Bureau of Statistics, 81 1 1 .0. and Go8 universities. Note: the effort amounts include estimates of university staff salaries 
apportioned to type of research activity. 


Thus there is a reasonable alignment between the functions of Go8 universities and the research- 
related indicators employed by the major world university rankings. Yet, as we have seen, Australia's 
leading universities, while ranking relatively well at present, are being challenged by the rapid rise 
of universities in Asia and elsewhere. Goncurrently, Australia's investment in research is in drift. 

Eigure 9 shows a trend decline in the basic research share of the total higher education research 
effort in Australia over the last 40 years. Whereas in 1 969 basic research represented 76.7% of 
university research in Australia by 201 0 it had fallen to 45.2%. Over the same period applied 
research increased its share from 1 9.7% to 46.7%, making it the dominant type of activity. To some 
extent, the change in reported types of research activity reflect changes to labelling, particularly 
as researchers have been reguired to relate their research grant proposals to socio-economic 
objectives or national priorities as a condition of funding. 
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Figure 9. Trends in shares of research by type of activity, Australia, 1 969-201 0 



Source: Australian Bureau of Statistics, 81 1 1.0, various issues. 


Nonetheless, the apparent trend decline in the share of basic research raises the question: 'what 
is the appropriate balance between basic and applied research in national higher education 
systems'? Australia now has less than 50% of higher education research defined as basic, including 
'strategic basic' (or in Frascati Manual terminology'oriented'research). Apart from the research of 
Medical Research Institutes, the large share of research in publicly funded research agencies (e.g. 
CSIRO) and private institutions and firms is application oriented. Australia has a much higher level 
of reliance on its universities than most other countries for its basic research. Even so, the basic 
research share of the total research effort of US academic institutions has remained reasonably 
stable at around three quarters, 76.7% in 1 970 and 74.6% in 2009 (NSF, 2012). It is not self-evident 
that Australia should continue to reduce its basic research effort. 

Importantly, the position of Australia's leading universities in the world university rankings, and 
especially those rankings which set a high bar in respect of research output quality, is primarily 
a function of capacity to undertake long-term fundamental research which advances knowledge 
on a global basis. 

Governments around the world increasingly view research funding as a purchase of technical 
information needed to further some specific goal rather than as a consequence of public policy 
aimed at creating educated researchers or new understandings for the long-term good of the 
nation (Armstrong, 2003). We are seeing a convergence internationally in a public policy emphasis 
on the utility of publicly-funded research, its demonstrable 'impact' and commercialisability. 

It is one thing to draw on impact exemplars as a basis for communicating to the lay community 
the benefits of their investment in research. It is quite another thing to limit research investment 
to research oriented only to a demonstrable benefit. It is limiting to do so, not least because the 
actual uses of research are not always immediately evident and the applications do not always 
turn out to serve the purposes for which the research was initially intended. Even worse is to 
spurn research that has no obvious immediate utility as not being worth funding. A proper sense 
of balance is necessary. Pressure for short-term instrumentalist outcomes from research can 
reduce incentives for speculative and path-breaking research. Indulgent long-term research can 
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miss opportunities for solving pressing social, environmental or economic problems. A narrow 
disciplinary focus risks forfeiting the great potential for creative ideas and new insights at the 
intersections between disciplines. 

In their reflective essay of 2008, What are universities for?, published by the League of European 
Research Universities, Geoffrey Boulton and Colin Lucas offer a corrective to the narrow and 
immediate instrumentalism of government policies in many countries. They note a "growing 
tendency to see universities as sources of highly specific benefits... particularly marketable 
commodities for their customers, be they students, business or the state."They suggest that 
research universities are able to make such contributions because they deal with the universality 
of knowledge: 

"They seek to understand that which we do not understand; they seek to explain 
complexity; they seek to discover that which is hidden from us. They seek to establish what 
is common to all of us and what distinguishes us each from another or each group from 
another. These things are common to the whole of university endeavour whatever the 
discipline. They are not 'academic' in the pejorative sense of the word, but are of profound, 
practical utility. They are the foundation upon which the university enterprise rests and upon 
which its significance for society is built" (Boulton & Lucas, 2008). 

Hence they argue that governments should respect the essential core of the research university 
and not act to erode or circumscribe it. This is not a novel reminder. As Derek Bok observed in 
1 990, universities "help in but do not determine" outcomes such as effective corporate governance, 
sound financial regulation, competent government, effective schools, improved health or reduced 
poverty. He cautioned that "we will debase our academic institutions and the work they do if we 
think of them merely or even primarily as means rather than ends" (Bok, 1 990). 

Debates over research orientations have been going on fruitlessly for more than half a century. 
Divisions have been evident within the academy, with scientists in the natural disciplines tending 
to regard basic research as intellectually'pure'and'deep'while Engineering researchers have 
tended to regard applied research as 'real'and 'worthwhile'. In practice, many of the former 
distinctions between basic and applied research have blurred. In the lay community, the traditional 
justifications for public funding of basic research have come under guestion, partly through the 
'democratisation'and 'commercialisation' of science - the challenge to researcher autonomy and 
authority and the demand for citizen participation, and the expectation of direct and immediate 
outcomes of research for product and process innovation. 

Vanevaar Bush argued that as basic research has no direct commercial application it will not get 
funding from industry, but as it produces scientific capital for the nation it deserves public support 
(Bush, 1 945). Conversely, applied research should be funded by industry not government. The Bush 
advocacy asserted a linear relationship from basic to applied research, suggesting that the guality 
of the underlying research is a precondition of applicability: technical developments are ultimately 
based on theoretical advances. The Erascati manual defines basic research in the following 
terms: "Basic research is experimental and theoretical work undertaken to primarily acguire new 
knowledge of the underlying foundations of phenomena and observable facts" (OECD, 1 994). 

A series of dichotomous descriptors has arisen in the course of debate, such as the following: 

• Basic research is performed without regard to practical ends whereas applied research is driven by 
finding solutions to practical problems. Yet basic research can yield surprising practical benefits 
(e.g. global positioning systems derived from research into the theory of relativity) and basic 
problems can arise in the process of exploring solutions to practical challenges, e.g. immersion 
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of materials in deep water environments. Developments in information and communications 
technology speed up the processes of discovery to exploitation, e.g. in biotechnology. 

• Basic research is general, applied research is contextual. Yet many breakthroughs in basic science 
have solved context-specific problems and insights derived from applied research have 
broadened theoretical understandings. 

• Basic research asks why? Applied research asks how? Yet for some, such as astronomers 
researching deep space, the answers to how we came into being produce the answers 
to why we are. 

• Basic is path-breaking, applied research is path-following. Yet, fundamental discoveries can 
be made in attempts to solve practical problems as basic problems are set and solved in 
application settings (Gibbons, et al. 1994). 

• Basic research is curiosity-driven or researcher initiated, whereas applied research is problem-driven 
or user-initiated. Yet, the initial intent of research does not necessarily constrain its conduct. 

• Basic research is long-term whereas applied research is short-term. Yet attempts to find cures for 
cancers and dementia have taken decades. 

In an endeavour to accommodate the overlap between basic and applied research, Stokes (1 997) 
proposed a new category of "use-inspired basic research" based on the aim of research spanning 
a guest for fundamental understanding and considerations of use (see Figure 1 0). 


Figure 10. Pasteur's Quadrant 


Considerations of Use 

Low 

High 

Pure Basic 
Research (Bohr) 

Use-Inspired Basic 
Research (Pasteur) 


Pure Applied 
Research (Edison) 


Stokes, 1 997 


The Stokes' approach is reflected in Australia's categories of research expenditure, including 'pure 
basic', 'strategic basic', and 'applied'. Even so, these categories relate to the intent of research rather 
than its content or processes or results. A plurality of approaches to research is necessary and 
patient expectations are reguired with regard to research outcomes. A wide definition of research 
can accommodate a diversity of research orientations: 

"In the broadest sense of the word, the definition of research includes any gathering of data, 
information and facts for the advancement of knowledge" (Shuttleworth, 2008). 
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Conclusion 


The current main world university rankings broadly group the leading research universities of 
nations. Australia's Go8 universities are generally within the top 250 ranked universities, with 
several institutions in the top 50-100 on some measures. This recognition is commendable, 
however imperfect the individual rankings may be. 

Use is made of rankings by prospective students, governments and universities themselves. There 
is not always a good alignment between the purposes for which rankings are designed and the 
ways they are used. 

All of the rankings are deficient methodologically, and some are seriously flawed. Considerable 
work needs to be done to overcome the major deficiencies in ranker methods, especially those 
reliant on reputational surveys and proxy indicators, to give them greater statistical validity and 
reliability and better representation of disciplinary fields and cross-disciplinary work. 

Because of the complexity of universities their assignment to ranks will always give a false sense 
of precision. It would be more appropriate for universities to be banded on the basis of their 
performance in relation to sets of criteria rather than ranked one after another. 

Because of the diversity of universities in terms of what and how they teach, research and 
otherwise engage with their communities, the evaluative criteria need to be appropriate to their 
varying roles and activities. The criteria that are relevant for research-intensive universities are not 
as suitable for universities that do not research in all areas they teach. 

For universities and other higher education providers whose functions are less research related, 
the current suite of world rankings are misleading for students and others and dangerous, neither 
appreciating what different institutions do and what they stand for, nor providing relevant 
references for their improvement. Indeed, the international exposure of the rankings - and all that 
goes with that in terms of branding for graduate opportunities, student recruitment, investment, 
partnering and alumni support - can press some universities to distort their missions and dilute 
the strengths they have and need to develop for serving their communities. 

For those institutions in particular a more diverse and balanced set of comparative indicators 
needs to be developed with a focus on rating their educational and service strengths, and by 
field of education as well as on a whole of institution basis. This will reguire the development 
of indicators related to teaching and learning, diverse research orientations, and community 
engagement functions. 

The current, mostly commercial rankings are filling a void, given the proliferation of data and 
marketing hype alongside the failure of universities themselves to adequately inform the 
communities that sustain them. Unless the university sector takes the initiative to develop 
appropriate and accessible ways of explaining their contributions, their directions will be 
increasingly determined by values set by others. One option is for groups of like institutions 
internationally to develop sets of criteria related to their missions in common, or at least to work 
with rating and ranking agencies to build typologies for comparative assessment. 

Meanwhile, for Australia's leading universities, it is no time for complacency. Their positions are 
challenged by the rapid rise of universities in Asia and elsewhere. Additionally they face three main 
public policy threats: under-investment; dilution of investment; and narrow instrumentalism of 
support. 

The first threat is one of under-investment in competitive research grants, research infrastructure. 
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and talent development - doctoral and postdoctoral students and early career researchers. 
Australian Government funding for research, research infrastructure and research training has 
plateaued and started to decline in recent years. There is currently no forward provision for real 
growth in any of the research funding programs. Without adeguate resource inputs Australia's best 
will not be able to stay with the pace of the world's best. 

The second threat is that of dilution of investment - a tendency to spread available resources 
across all universities, typically on a formulaic basis, and to avoid the concentration necessary for 
achieving the capacity to do internationally-recognised breakthrough research. 

The third threat is that of narrow instrumentalism - a bias against basic research, and a preference 
for applied research with a demonstrable impact and immediate commercial pay off 

A combination of under-investment, social levelling and narrow instrumentalism in policy 
for higher education and university research is inimical to sustaining excellence for those 
universities whose mission is liberal and professional education of high guality in the context of 
internationally-referenced basic research and nationally-applicable applied research. If these three 
threats were to be realised in Australian public policy - and they are imminent - we will not be 
celebrating for much longer the position of our leading universities in the world league. 
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Appendix 1 . The construction of rankings 

There are major differences in the methods used by the five main rankings. These differences can 
be seen in the definitions of what constitutes quality, in the criteria and indicators used to measure 
quality, in the measurement processes and in the presentation format. 

ARWU 

The ARWU is focused on research metric-based indicators: 

• Alumni of an institution winning Nobel Prizes and Fields Medals (10%) 

• Staff of an institution winning Nobel Prizes and Fields Medals (20%) 

• Flighly cited researchers in 21 broad subject categories (20%), 

• Papers indexed in Science Citation Index-expanded and Social Science Citation Index (20%) 

• Papers published in Nature and Science (20%) 

• Per capita academic performance of an institution (1 0%) 

The total number of the alumni of an institution winning Nobel Prizes and Fields Medals. Alumni 
are defined as those who obtain bachelor. Master's or doctoral degrees from the institution. 
Different weights are set according to the periods of obtaining degrees. The weight is 1 00% for 
alumni obtaining degrees in 2001-2010, 90% for alumni obtaining degrees in 1991-2000, 80% for 
alumni obtaining degrees in 1 981 -1 990, and so on, and finally 1 0% for alumni obtaining degrees 
in 1911 -1 920. If a person obtains more than one degrees from an institution, the institution is 
considered once only. 

The total number of the staff of an institution winning Nobel Prizes in Physics, Chemistry, Medicine 
and Economics and Fields Medal in Mathematics. Staff are defined as those who work at an 
institution at the time of winning the prize. Different weights are set according to the periods of 
winning the prizes. The weight is 1 00% for winners after 201 1 , 90% for winners in 2001 -2010, 80% 
for winners in 1 991-2000, 70% for winners in 1 981 -1 990, and so on, and finally 1 0% for winners in 
1921-1930. If a winner is affiliated with more than one institution, each institution is assigned the 
reciprocal of the number of institutions. For Nobel prizes, if a prize is shared by more than one 
person, weights are set for winners according to their proportion of the prize. 

The number of Flighly Cited Researchers in 21 subject categories includes those individuals that 
are the most cited within each category as reported by Thomson Reuters. Being acknowledged 
by Thomson Reuters as a Flighly Cited Researcher means that an individual was among the 250 
most-cited researchers in a defined discipline within a specific time-period based on analysis of 
papers covered in Web of Science from 1 981 -2008. If a Flighly Cited Researcher has two or more 
affiliations, they were asked to estimate their weights (or number of weeks) for each affiliation. 
More than 2/3 of those multi-affiliated Flighly Cited Researchers provided such estimations and 
their affiliations receive the weights accordingly. For those who did not answer, their first affiliation 
is given a weight of 84% (average weight of the first affiliations for those who replied) and the rest 
affiliations share the remaining 1 6% equally. 

The total number of papers indexed in Science Citation Index-Expanded and Social Science 
Citation Index in 201 1 accounted for 20%. Only publications of 'Article'and 'Proceedings Paper' 
types are considered. When calculating the total number of papers of an institution, a special 
weight of two was introduced for papers indexed in the Social Science Citation Index. 
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The number of papers published in Nature and Science between 2007 and 201 1 accounted 
for 20%. To distinguish the order of author affiliation, a weight of 100% was assigned for 
corresponding author affiliation, 50% for first author affiliation (second author affiliation if the 
first author affiliation was the same as corresponding author affiliation), 25% for the next author 
affiliation, and 1 0% for other author affiliations. Only publications of 'Article' and 'Proceedings Paper' 
types were considered. 

In addition the bibliometric indicators were included as part of the per capita indictor in which 
weighted scores of five indicators were divided by the number of full-time eguivalent academic 
staff The per capita indicator amounts to 1 0% of the ranking score. 

Both the Highly-cited and publications in Nature and Science Indicators are strongly lagged 
indictors with the former based on citation data from 1 981 -2008 and the latter based on 
publications between 2007 and 201 1 . 

The methodology has the purported advantage that the data is metric based and does not rely on 
data provided by the universities themselves or on surveys. The focus is clearly on research and not 
on teaching and learning. 

Times Higher Education -Thomson Reuters 

The THEThomson Reuters 2012-2013 ranking was derived (as in 201 1-12) from 13 separate 
indicators grouped under five categories: Teaching (reputational survey 15%, a range of student/ 
staff ratios and income per academic 1 5% ), research (reputational survey 1 5% and a range of 
income and publication measures 1 5%), citations (30%), international outlook (ratios of international 
to domestic staff and international students to domestic students, research measure 7.5%), industry 
income (research income from industry scaled against the number of academic staff 2.5%). 


Overall indicator 

Individual indicators 

Percentage 

weightings 

Industry Income - innovation 

Research income from industry (per academic staff) 

2.5% 

International outlook (staff 

Ratio of international to domestic staff 

2.5% 

students, research) 

Ratio of international to domestic students 

2.5 


Proportion of internationally co-authored research papers 

2.5% 

Teaching - the learning 

Reputational survey (teaching) 

15% 

environment 

PhDs awards per academic 

6% 


Undergrad, admitted per academic 

4.5% 


Income per academic 

2.25% 


PhDs/undergraduate degrees awarded 

2.25% 

Research - volume, income and 

Reputational survey (research) 

18% 

reputation 

Research income (scaled) 

6% 


Papers per research and academic staff 

6% 

Citations - research influence 

Citation impact (normalised average citation per paper) 

30% 


The reputational component of the rankings is derived from an Academic Reputation Survey of 
'representative' academic staff around the world conducted by Thomson Reuters. 

Data for citation impact (measured as a normalised average citation per paper) comes from 
Thomson Reuters' database over a five year period. 

The THEThomson Reuters ranking method was changed in 2010. Earlier rankings used the ranking 
system produced by QS (see below). The ranking method changed again from 201 1 to 201 2 with 
the inclusion of the research indicator in international outlook, a reduction in the weighting for 
citations and other minor changes. 


PAGE 56 


WORLD UNIVERSITY RANKINGS: AMBIGUOUS SIGNALS | Go8 BACKGROUNDER 30 






QS World University Rankings 

QS uses a ranking based on Academic peer review (40%), recruiter review (1 0%), Facuity student 
ratio (20%), Citations per facuity (20%) and internationai orientation (10%). 

The Academic Peer Review is a survey which asks active academics across the worid about the top 
universities in fieids they know about. The recruiter review sampies recruiters who hire graduates 
on a giobai or significant nationai scaie. 

QS previously used citations data from Thomson Reuters for its 2004 to 2007 rankings but now 
uses data from Scopus (Elsevier). The total number of citations for a five-year period is divided by 
the number of academic staff in a university to yield the score for this measure, which accounts for 
20 per cent of a university's possible score in the Rankings. 

For major world universities, the Scopus and Thomson Reuters databases capture more or less the 
same publications and citations. For less mainstream institutions, Scopus has more non-English 
language and smaller-circulation journals in its database. But as the papers there are less heavily 
cited, this can also mean fewer citations per paper for the universities that publish in them. 

International orientation is derived from measures intended to capture their internationalism: 
five percent from their percentage of international students, and another five percent from their 
percentage of international staff 

The QS ranking, although including both teaching and research measures, is heavily based on 
reputational surveys and as such has been criticised for the validity of its ranking system. 

QS Stars 

QS Stars is described as a system that evaluates universities worldwide using a rating method. 
Universities are awarded a rating of one to five stars against eight criteria; Research Quality, 
Teaching Quality, Graduate Employability, Infrastructure, Internationalisation, Innovation & 
Knowledge Transfer, Third Mission and Specialist Subject Criteria. Each criterion has its own 
indicators, weightings. 

A typical Four Stars university described as "highly international, demonstrating excellence in both 
research and teaching. The institution provides an excellent environment for students and faculty." 
A typical Five Stars university is described as "generally world class in a broad range of areas, enjoys 
a high reputation and has cutting edge facilities and internationally renowned research and 
teaching faculty. A typical five-star plus institution is "not just world-class, but an elite destination 
to which the very best students and faculty worldwide will aspire. Its brand name will transform 
the resume of anyone connected with it. Five Stars -i- can apply egually to the world's foremost 
comprehensive and specialist institutions." 

Three Australian universities are recorded as having 5stars-i- ratings: Monash University, The 
University of New South Wales, and The University of Queensland. Seven are recorded as having 
Sstars: Deakin; Newcastle; UniSA; UTS, QUT, RMIT, and Wollongong. Two are recorded with 4stars: 
Swinburne andTasmania. 

QS Stars ratings may be obtained by a university paying for a special QS audit at a cost for the 
audit and a three-year licence of US$30,400 (Jibbins, 2012). Some universities - Cambridge 
and Flarvard among them - have not had to pay for the accreditation process for their five-star 
rating as their award is based on publicly available information. The main market for QS stars are 
institutions not ranked highly on the basis of publicly available metrics. 
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Head of the QS Intelligence Unit Ben Sowter, is reported as saying: 

"In global terms, viewing the guality of world universities through a frame of reference 
defined by the likes of Cambridge and Harvard has limitations. With universities all over 
the world charging substantial fees to international students, an objective means of 
differentiating between the guality of the services they provide is indispensable. QS Stars 
allow students to make more informed choices, while also helping universities to recruit 
more effectively by providing independent verification of the areas in which they excel" 

(Jibbins, 201 2). 

That is nothing short of a marketing spiel along the lines that 'if you can't cut it in the objective 
measurement stakes then buy our ratings and you will more than recoup the costs through 
growth in student fee revenues'. It is an attractive business model @$35,000 for potentially 1 7,000 
institutions = $595 million per year. While some universities are opting to gain a QS star rating for 
marketing purposes many are (yet?) not, including those that would be likely to be rated highly 
and those not so highly rated. It will be interesting to see whether a divide will emerge between 
those who pay to be rated and those that do not. 

Leiden Ranking 

The Leiden Ranking 201 1/2012 is based on publications inThomson Reuters'Web of Science 
database in the period 2005-2009. Qnly publications (articles, letters, and reviews) in the sciences 
and the social sciences are included. Publications in the arts and humanities are excluded because 
in these domains the bibliometric indicators do not have sufficient accuracy. As a result the Leiden 
ranking is a measure of research in the physical and social sciences and with no metrics relating to 
teaching and learning 

The Leiden Ranking includes the following bibliometric indicators: 

• Mean citation score (MC5). The average number of citations of the publications of a university. 

• Mean normalized citation score (MNC5). The average number of citations of the publications of 
a university, normalized for field differences, publication year, and document type. An MNCS 
value of two for instance means that the publications of a university have been cited twice 
above world average. 

• Proportion top 10% publications The proportion of the publications of a university 

that, compared with other similar publications, belong to the top 1 0% most freguently cited. 
Publications are considered similar if they were published in the same field and the same 
publication year and if they have the same document type. 

Citations are counted until the end of 201 0 in the above indicators. Author self citations are excluded. 

In addition a range of bibliometric indicators measuring research collaboration are included: 

• Proportion collaborative publications (PP^^nJ.Jhe proportion of the publications of a university 
that have been co-authored with one or more other organizations. 

• Proportion international collaborative publications proportion of the publications 

of a university that have been co-authored by two or more countries. 

• Mean geographical collaboration distance (MGCD). The average geographical collaboration 
distance of the publications of a university, where the geographical collaboration distance of a 
publication eguals the largest geographical distance between two addresses mentioned in the 
publication's address list. 
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• Proportion long distance collaborative publications (PP^,ggoi,J- The proportion of the publications 
of a university that have a geographical collaboration distance of more than 1 000 km. 

The Leiden Ranking can be adjusted to take into account different ways of counting joint 
publications and adjusting for non-English language publications. 

Webometrics 

The Ranking Web or Webometrics has been produced since 2004 and is revised every six months 
by the Cybermetrics Lab of the Spanish National Research Council (CSIC). It is derived from 
information about the performance of universities from all over the world based on their web 
presence and impact. Since 2008 the portal has also included webometrics rankings for research 
centres, hospitals, repositories and business schools. The ranking aims to promote academic web 
presence and support Open Access initiatives for increasing the transfer of scientific and cultural 
knowledge generated by the universities to the wider society. 

Webometrics uses link analysis rather than citation analysis or reputational surveys for guality 
evaluation. Webometrics log-normalise the variables before combining according to a ratio 1 :1 
between activity/presence and visibility/impact groups of indicators. The July 201 2 composite 
indicator comprises: 

• Presence (20%). The total number of webpages hosted in the main webdomain (including all 
the subdomains and directories) of the university as indexed by the largest commercial search 
engine (Google). 

• Impact (50%). Content guality is evaluated through a "virtual referendum", counting all the 
external inlinks that the University webdomain receives from third parties. The link visibility data 
is collected from Majestic SEO and ahrefs. Both use their own crawlers, generating different 
databases. The indicator is the product of number of backlinks and the number of domains 
originating those backlinks. The maximum of the normalised results is the impact indicator. 

• Openness (15%). This indicator takes into account the number of rich files (pdf doc, docx, ppt) 
published between 2007 and 201 1 in dedicated websites according to the academic search 
engine Google Scholar. 

• Excellence (1 5%). This indicator counts scientific output within the 1 0% most cited papers in 
their respective scientific fields. 

HEEACT rankings 

EIEEAGT uses the following criteria: 

• Research productivity (weighed 20%) -The number of published articles of the last 1 1 years 
(10%) and the number of articles of the current year (10%). 

• Research impact (weighed 30%) - Number of citations of the last 1 1 years (10%), the number of 
citations of the last two years (10%), and the average number of citations of the last 1 1 years (10%). 

• Research excellence (weighed 40%) - The h-index of the last two years (20%), the number 
of highly-cited papers (1 5%), and the number of articles of the current year in high-impact 
journals (15%). 

Quantitative data is drawn from Science Gitation Index (SGI) and Social Sciences Gitation Index 
(SSGI). The data is normalized by faculty number to account for different institution sizes. The 
indicators used in this methodology highly emphasize research guality (80% of the performance 
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score) and short-term research performance (55% of the score). 

The HEEACT rankings were terminated in 201 1. The Taiwanese education authority has announced 
that the Government wiii no ionger support rankings by Eiigher Education Evaiuation and 
Accreditation Councii ofTaiwan. 

SIR World Report 

The SiR Worid Report 2012 inciudes 3290 research institutions from 1 06 countries. The indicators 
inciude: 

• Output. Totai number of documents pubiished in schoiariy journais indexed in Scopus. 

• International Collaboration, institution's output ratio produced in coiiaboration with foreign 
institutions. The vaiues are computed by anaiysing an institution's output whose affiiiations 
inciude more than one country address. 

• Normalised Impact. The vaiues (in %) show the reiationship between an institution's average 
scientific impact and the worid average set to a score of 1, i.e. a Ni score of 0.8 means the 
institution is cited 20% beiow worid average and 1 .3 means the institution is cited 30% above 
average. 

• High Quality Publications. Ratio of pubiications that an institution pubiishes in the most 
infiuentiai schoiariy journais of the worid; those ranked in the first guartiie (25%) in their 
categories as ordered by SCimago Journai Rank SJR indicator. 

• Specialisation Index. The Speciaiisation index indicates the extent of thematic concentration / 
dispersion of an institution's scientific output. Vaiues range between 0 to 1 , indicating generaiist 
vs.speciaiised institutions respectiveiy. This indicator is computed according to the Gini index 
for the nationai economy. 

• Excellence Rate. The Exceiience Rate indicates the amount (in %) of an institution's scientific 
output that is inciuded into the set of the 1 0% of the most cited papers in their respective 
scientific fieids. 

• Scientific Leadership. Leadership indicates the number of papers in which the corresponding 
author beiongs to the institution. 


PAGE 60 


WORLD UNIVERSITY RANKINGS: AMBIGUOUS SIGNALS | Go8 BACKGROUNDER 30 



Indicators of Australia's Group of Eight Universities group of eight 
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